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Abstract 


Building  on  an  earlier  small-sample  interview  study  (Derbentseva,  McLellan,  &  Mandel,  2011), 
this  report  describes  the  findings  of  a  focus  group  study  with  members  of  the  Canadian  intelligence 
community.  The  present  study  had  both  a  larger  and  more  diverse  sample  of  intelligence 
practitioners  than  the  earlier  study.  Four  focus  group  discussions  were  conducted  to  explore  human 
capability  challenges  within  the  broader  Canadian  community.  The  study  also  explored  how 
behavioural  science  research  might  help  the  intelligence  community  deal  with  the  identified 
challenges.  Results  showed  a  wide  range  of  issues  and  challenges  identified  within  the  focus 
groups.  Issues  such  as  coordination  and  information  sharing  within  the  community, 
professionalization  and  the  need  for  better  career  paths  emerged  as  important  challenges.  Educating 
consumers  about  intelligence  analysis,  clarifying  the  relationship  between  consumers  and 
producers  of  analytic  products  and  research  related  to  the  tools,  techniques  and  the  practice  of 
intelligence  also  received  considerable  attention.  Overall,  this  study  documents  an  expanded  set  of 
issues  and  challenges  facing  intelligence  personnel,  strong  evidence  of  the  potential  contribution 
that  future  research  can  make  to  alleviating  current  challenges  within  the  intelligence  community, 
and  a  detailed  list  of  potential  research  opportunities  for  behavioural  science  research  (and  other 
types  of  research)  to  support  intelligence  capability. 


Human  Capability  Challenges  in  Intelligence  Analysis 


Page  i 


HUMAUSYSTEMS 


Resume 


Fonde  sur  une  etude  anterieure  d’un  petit  echantillonnage  d’entrevues  (Derbentseva,  McLellan  & 
Mandel,  2011),  le  present  compte  rendu  decrit  les  resultats  obtenus  d’une  etude  menee  par  un 
groupe  de  consultation  avec  des  membres  de  la  collectivite  canadienne  du  renseignement.  Cette 
etude  renferme  un  echantillonnage  plus  important  et  plus  varie  de  praticiens  du  renseignement  que 
F  etude  precedente.  Quatre  groupes  de  consultation  ont  discute  des  problemes  de  capacite  humaine 
au  sein  de  la  collectivite  canadienne  dans  son  ensemble.  L’ etude  a  egalement  analyse  de  quelle 
fagon  la  recherche  en  science  du  comportement  peut  aider  la  collectivite  du  renseignement  a 
aborder  les  problemes  identifies.  Les  resultats  ont  montre  une  vaste  gamme  d’enjeux  et  de 
problemes  identifies  par  les  groupes  de  consultation.  Des  questions  comme  la  coordination  et  le 
partage  de  f  information  au  sein  de  la  collectivite,  la  professionnalisation  et  le  besoin  de  meilleurs 
parcours  professionnels  sont  apparues  importantes.  L’education  des  consommateurs  sur  l’analyse 
du  renseignement,  la  clarification  des  rapports  entre  les  consommateurs  et  les  producteurs  de 
produits  analytiques  ainsi  que  la  recherche  associee  aux  outils,  aux  techniques  et  a  la  pratique  du 
renseignement  ont  egalement  fait  l’objet  d’une  attention  particuliere.  Dans  l’ensemble,  cette  etude 
porte  sur  un  vaste  ensemble  d’enjeux  et  de  problemes  auxquels  le  personnel  du  renseignement  est 
confronte,  sur  une  preuve  solide  de  la  contribution  eventuelle  que  peut  apporter  la  recherche  future 
pour  amoindrir  les  problemes  actuels  au  sein  de  la  collectivite  du  renseignement  et  sur  une  liste 
detaillee  des  possibility  de  recherche  en  science  du  comportement  (et  autres  types  de  recherche)  a 
l’appui  de  la  capacite  du  renseignement. 
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Executive  Summary 

Capability  Challenges  in  the  Human  Domain  for 
Intelligence  Analysis:  Report  on  Community-Wide 
Discussions  with  Canadian  Intelligence  Professionals 

Barbara  D.  Adams;  Michael  H.  Thomson;  Humansystems  Incorporated; 
Natalia  Derbentseva;  David  R.  Mandel;  DRDC  Toronto  CR2011-182;  Defence 
R&D  Canada  Toronto;  March  2012 

Background  -  Intelligence  plays  a  vital  role  in  defence  and  security.  It  is  important  for  the 
intelligence  community  to  take  stock  of  its  capability  challenges  in  various  domains  and  to  find 
ways  to  adapt  to  meet  new  challenges.The  present  study  was  aimed  at  identifying  human  capability 
challenges  in  the  Canadian  intelligence  community,  with  a  particular  view  toward  identifying  how 
behavioural  science  might  help  to  respond  effectively  to  these  challenges.  To  this  end,  four  focus 
groups  were  conducted  with  members  of  the  Canadian  intelligence  community.  This  research 
builds  on  an  earlier  small-scale  interview  study  with  managers  of  intelligence  analysts 
(Derbentseva,  McLellan,  &  Mandel,  2011).  The  present  study  not  only  includes  a  larger  sample,  it 
also  includes  representation  from  a  much  wider  range  of  organizations  and  roles  (e.g.,  analysts, 
managers,  and  trainers)  from  the  community. 

Method  -  Twenty-three  subject  matter  experts  from  eight  organizations  within  the  intelligence 
community  (CBSA,  CFSMI,  CSIS,  CDI,  ITAC,  PCO,  PSC,  and  RCMP)  participated  in  one  of  four 
focus  groups.  This  research  was  conducted  in  two  phases.  Phase  1  adopted  a  “brainstorming” 
approach  in  which  participants  identified  topics  and  issues  that  they  perceived  as  being  of 
significant  concern.  The  results  of  this  elicitation  process  were  then  summarized  in  ten  broad 
categories  and  related  subtopics  and  issues.  In  Phase  2,  focus  groups  validated  and  expanded  the  set 
of  issues  identified  during  the  Phase  1 .  Participants  in  both  phases  also  rated  the  potential  value  of 
research  exploring  each  the  issues  generated  during  discussion. 

Results-A  wide  range  of  issues  and  challenges  were  identified  by  participants,  including 
understanding  the  analyst’s  motivation;  improving  recruitment  practices;  developing  processes  and 
tools  for  evaluation  of  intelligence  processes  and  products;  educating  consumers  about  the 
intelligence  process  and  products;  improving  communication  of  intelligence  through  improving  the 
communication  of  uncertainty  and  development  of  information  visualization  techniques; 
documenting  existing  tools  and  practices  and  evaluating  their  effectiveness;  promoting  collegial 
collaboration  through  informal  networking;  understanding  the  role  of  mentoring  and  its  impact  on 
intelligence  personnel;  the  lack  of  coordination  and  information  sharing  within  the  community;  the 
need  for  community-wide  coordination  of  joint  efforts  and  an  analyst  secondment  programme;  the 
need  for  professionalization  of  intelligence  analysis;  and  the  need  to  expand  career  paths  available 
to  analysts.  Overall,  issues  at  the  community-wide  level  received  greatest  attention  during  the 
discussions. 

Significance-The  study  documents  a  wide  range  of  human  capability  challenges  in  the  intelligence 
domain.  The  information  garnered  from  this  study  could  profitably  be  used  by  both  scientific 
organizations,  such  as  DRDC,  and  defence  and  security  organizations,  such  as  those  from  which 
the  participants  were  drawn,  to  chart  a  roadmap  for  science  and  technology  in  support  of  the 
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intelligence  community.  In  particular,  the  results  of  this  study  could  inform  efforts  to  develop  a 
vision  for  how  DRDC  might  contribute  to  intelligence  capability  now  and  into  the  future. 

Future  Research-A  detailed  list  of  potential  research  opportunities  was  identified  and  feedback 
was  also  elicited  from  participants  to  inform  a  survey  to  be  undertaken  with  a  broad  cross-section 
of  organizations  within  the  Canadian  intelligence  community  in  the  2012-13  fiscal  year.  This 
survey  will  further  examine  the  human  challenges  in  intelligence  production  in  order  to  help  set 
priorities  for  behavioural  science  research  undertaken  by  DRDC  Toronto  and  possibly  other  DRDC 
centres. 
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Capability  Challenges  in  the  Human  Domain  for 
Intelligence  Analysis:  Report  on  Community-Wide 
Discussions  with  Canadian  Intelligence  Professionals 

Barbara  D.  Adams;  Michael  H.  Thomson;  Humansystems  Incorporated; 
Natalia  Derbentseva;  David  R.  Mandel;  DRDC  Toronto  CR2011-182;  Defence 
R&D  Canada  Toronto;  December  2011 

Contexte  -  Le  renseignement  joue  un  role  essentiel  en  matiere  de  defense  et  de  securite.  11  est 
important  que  la  collectivite  du  renseignement  repertorie  les  problemes  de  sa  capacite  dans  divers 
domaines  et  trouve  des  moyens  de  s’ adapter  afin  de  satisfaire  aux  nouveaux  enjeux.  La  presente 
etude  visait  a  identifier  les  problemes  de  capacite  humaine  au  sein  de  la  collectivite  canadienne  du 
renseignement,  tout  en  portant  une  attention  particuliere  a  la  faijon  dont  la  science  du 
comportement  peut  aider  a  regler  efficacement  ces  problemes.  A  cette  fin,  quatre  groupes  de 
consultation  ont  discute  avec  des  membres  de  la  collectivite  canadienne  du  renseignement.  Cette 
recherche  repose  sur  une  etude  anterieure  d’un  petit  echantillonnage  d’entrevues  avec  des 
gestionnaires  d’analystes  du  renseignement  (Derbentseva,  McLellan,  &  Mandel,  2011).  Non 
seulement  la  presente  etude  renferme  un  echantillonnage  plus  important,  mais  elle  contient  aussi  la 
representation  d’une  plus  vaste  gamme  d’ organisations  et  de  roles  (p.  ex.,  analystes,  gestionnaires, 
formateurs)  de  la  collectivite. 

Methode  -  Vingt-trois  specialistes  en  la  matiere,  issus  de  huit  organisations  de  la  collectivite  du 
renseignement  (ASFC,  ERMFC,  SCRS,  CRD,  CCT1,  BPC,  SPC  et  GRC),  ont  participe  a  Fun  des 
quatre  groupes  de  consultation.  La  recherche  s’ est  deroulee  en  deux  etapes.  Lors  de  la  premiere 
etape,  on  a  adopte  une  approche  dite  de  «  remue-meninges  »  au  cours  de  laquelle  les  participants 
ont  identifie  des  sujets  et  des  questions  qu’ils  estimaient  importants.  Les  resultats  de  ce  processus 
de  definition  ont  ensuite  ete  resumes  a  l’interieur  de  dix  categories  ainsi  que  de  sous-sujets  et  de 
questions  connexes.  Au  cours  de  la  deuxieme  etape,  les  groupes  de  consultation  ont  valide  et  elargi 
F ensemble  des  questions  identifies  lors  de  la  premiere  etape.  Les  participants  des  deux  etapes  ont 
egalement  evalue  la  valeur  potentielle  de  la  recherche  en  examinant  chacune  des  questions  relevees 
lors  des  discussions. 

Resultats  -  Une  vaste  gamme  d’ enjeux  et  de  problemes  ont  ete  identifies  par  les  participants,  y 
compris  la  comprehension  de  la  motivation  des  analystes,  F  amelioration  des  pratiques  de 
recrutement,  Felaboration  de  processus  et  d’outils  devaluation  des  processus  et  des  produits 
associes  au  renseignement,  Feducation  des  consommateurs  sur  les  processus  et  les  produits 
associes  au  renseignement,  F  amelioration  de  la  transmission  du  renseignement  par  Fentremise  de 
la  transmission  de  F incertitude  et  Felaboration  de  techniques  de  visualisation  du  renseignement,  la 
documentation  de  pratiques  et  d’outils  existants  et  F  evaluation  de  leur  efficacite,  la  promotion 
d’une  collaboration  collegiale  par  Fentremise  d’un  reseautage  informel,  la  comprehension  du 
mentorat  et  de  son  incidence  sur  le  personnel  du  renseignement,  le  manque  de  coordination  et  de 
partage  de  F information  au  sein  de  la  collectivite,  le  besoin  d’une  coordination  des  efforts  conjoints 
a  l’echelle  de  la  collectivite  et  d’un  programme  d’affectation  d’analystes,  le  besoin  de 
professionnalisation  de  l’analyse  du  renseignement  et  la  necessity  d’elargir  les  parcours 
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professionnels  disponibles  pour  les  analystes.  Dans  l’ensemble,  les  sujets  a  l’echelle  de  la 
collectivite  ont  regu  une  attention  hautement  particuliere  lors  des  discussions. 

Portee  -L’etude  aborde  une  vaste  gamme  de  problemes  de  capacite  humaine  dans  le  domaine  du 
renseignement.  L’information  tiree  de  cette  etude  peut  etre  utilisee  a  profit  par  les  deux  organismes 
scientifiques,  comme  RDDC  et  les  organisations  chargees  de  la  securite  et  de  la  defense,  comme 
celles  d’ou  proviennent  les  participants  pour  elaborer  une  carte  de  science  et  de  technologie  a 
l’appui  de  la  collectivite  du  renseignement.  De  fagon  particuliere,  les  resultats  de  cette  etude 
peuvent  servir  aux  efforts  d’elaboration  d’une  vision  sur  la  fagon  dont  RDDC  peut  contribuer  a  la 
capacite  du  renseignement,  maintenant  et  dans  le  futur. 

Recherches  futures  -Une  liste  detaillee  de  possibilites  de  recherche  a  ete  dressee  et  des 
commentaires  ont  ete  definis  par  les  participants  pour  permettre  la  tenue  d’une  enquete  aupres  d’un 
echantillonnage  representatif  d’ organisations  de  la  collectivite  canadienne  du  renseignement  au 
cours  de  l’AF  2012-2013.  Cette  enquete  examinera  plus  en  profondeur  les  problemes  humains  dans 
la  production  du  renseignement  afin  d’ aider  a  etablir  des  priorites  pour  la  recherche  en  science  du 
comportement  entreprise  par  RDDC  Toronto  et  possiblement  par  d’autres  centres  de  RDDC. 
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1.  Introduction 


1.1  Background 

Governments  operate  in  a  dynamic  environment,  and  to  ensure  their  survival  and  successful 
functioning,  they  need  to  have  an  adequate  response  system  to  changes  and  threats  in  the 
environment.  Two  of  the  several  key  elements  of  such  a  response  system  are  the  awareness  of 
current  events  and  the  means  of  anticipating  future  developments,  both  of  which  are  supplied  to  the 
government  by  its  intelligence  organizations.  Although  intelligence  is  often  left  behind  the  scenes, 
it  plays  a  pivotal  role  in  state  decision  making  and  policy  development  (Davis,  2006).  Only  when 
something  goes  terribly  wrong,  as  with  the  September  11,  2001,  attacks  on  the  United  States  or 
misjudgement  of  the  Iraqi  weapons  of  mass  destruction  program,  the  intelligence  (or  its  failure) 
comes  into  the  forefront(Bruce  &  George,  2008).  Investigations  into  the  reasons  for  these  failures 
had  been  conducted  by  Canada’s  allies  (e.g.,  Butler,  Chilcot,  Inge,  Mates,  &  Taylor,  2004;  9/1 1 
Commission,  2004),  and  the  intelligence  community,  in  general,  received  greater  attention  from  the 
scholarly  community  as  (e.g.,  Campbell  &  Mandel,  2010;  Fischhoff  &  Chauvin,  2011;  George  & 
Bruce,  2008;  Lefebvre,  2004;  NRC  Report,  2011).  Clearly,  these  investigations  aim  to  provide 
support  to  the  intelligence  community,  to  ensure  the  effectiveness  of  the  intelligence  function,  and 
to  prevent  future  intelligence  failings. 

To  achieve  these  ambitious  goals,  it  is  critical  to  isolate  the  challenges  that  the  intelligence 
community  is  facing  and  to  identify  means  of  addressing  them.  Although  more  and  more  tools  and 
systems  have  been  developed  over  the  years  to  assist  intelligence  analysts  in  their  work,  the  key 
element  and  the  heart  of  the  intelligence  production  remains  the  human.  The  human  component  of 
intelligence  production  encompasses  many  aspects,  including  thinking  and  reasoning  of  individual 
analysts,  the  organization  and  tasking  of  intelligence  institutions,  the  relationship  between 
intelligence  consumer  and  intelligence  producer  including  communication  and  knowledge  transfer 
between  them,  and  relationships  among  different  organizations  in  the  intelligence  community. 
Therefore,  the  emphasis  on  understanding  human  capability  challenges  in  intelligence  production  is 
necessary. 

As  indicated  above,  the  interest  in  the  issues  of  intelligence  production  has  been  growing,  and  more 
scholarly  work  has  been  undertaken  on  the  subject  in  the  recent  years.  The  majority  of  this  work 
has  been  conducted  in  the  U.S.  and  the  U.K.  and  has  emphasized  issues  within  these  intelligence 
communities.  However,  the  Canadian  intelligence  community  has  not  received  as  much  attention 
and  there  is  very  little  open  literature  available  dedicated  to  the  issues  and  concerns  pertinent  to  the 
Canadian  community. 

The  study  presented  in  this  report  focuses  on  the  Canadian  intelligence  community  with  the  goal  of 
identifying  human  capability  challenges  relevant  to  the  Canadian  context.  The  present  study  is  an 
extension  of  an  earlier  interview  study  conducted  by  the  DRDC  research  team  (Derbentseva, 
McLellan,  &  Mandel,  2011).  In  2008,  the  DRDC  research  team  interviewed  managers  of 
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intelligence  analysts  from  two  organizations,  including  the  Chief  of  Defence  Intelligence  (CDI)  and 
the  International  Assessment  Staff  (IAS)1  of  Privy  Council  Office  (PCO).  These  interviews  focused 
on  the  challenges  in  intelligence  production,  however  the  sample  in  the  initial  interview  study  was 
fairly  constrained.  Although  there  are  many  similarities,  there  are  also  serious  differences  in 
emphasis  and  role  of  intelligence  analysts  and  managers  working  within  the  diverse  Canadian 
intelligence  community.  The  present  study  addresses  this  issue  by  including  a  larger  sample  of 
intelligence  professionals  in  order  to  capture  the  experiences  of  personnel  in  the  Canadian 
intelligence  community  from  a  broader  set  of  organizations. 

The  focus  groups  described  in  this  report  were  aimed  at  identifying  the  issues  and  challenges  facing 
personnel  working  within  the  intelligence  community,  with  a  view  toward  identifying  how 
behavioural  science  research  might  help  to  alleviate  these  challenges.  Feedback  was  also  elicited 
from  participants  to  inform  a  survey  to  be  undertaken  with  a  broad  cross-section  of  organizations 
within  the  Canadian  intelligence  community  in  the  2012-13  fiscal  year.  This  survey  will  further 
examine  the  human  challenges  in  intelligence  in  order  to  help  set  priorities  for  behavioural  science 
research  undertaken  by  DRDC  Toronto  and  possibly  other  DRDC  centres. 

The  report  is  organized  in  the  following  manner:  The  next  section,  section  2,  describes  the 
methodology  used  to  conduct  the  study;  section  3  reports  the  results  of  quantitative  assessments 
and  the  summary  of  the  discussions;  and  section  4  provides  research  implications  and  conclusions. 


1  Note  that  IAS  has  reverted  to  the  name  Intelligence  Assessment  Secretariat. 
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2.  Method 


2.1  Participants 

Participants  for  this  study  were  identified  by  personnel  within  the  intelligence  community  and  the 
professional  contact  networks  of  the  DRDC  Toronto  authors  of  this  report.  The  DRDC  Toronto 
authors  sent  an  invitation  by  email  to  the  prospective  participants,  requesting  their  voluntary 
participation  in  the  present  study.  The  letter  stated  that  DRDC  Toronto  is  conducting  a  multi-year 
research  project  that  aims  to  apply  behavioural  science  research  to  support  the  analytic  function  in 
intelligence.  The  proposed  study  sought  to  extend  an  initial,  small-sample  interview  study  of 
human  capability  challenges  in  the  intelligence  domain  to  a  wider  sample  from  the  Canadian 
intelligence  community.  The  letter  further  explained  that  the  research  team  planned  to  distribute  a 
survey  to  intelligence  analysts  and  managers  of  analysts  across  the  intelligence  community,  but  that 
as  a  preliminary  step,  the  research  team  would  like  to  solicit  input  from  analysts,  managers,  and 
educators  in  the  intelligence  community  regarding  potential  areas  of  investigation  for  the  study. 
Thus,  the  letter  explained,  the  research  team  was  now  organizing  a  series  of  half-day  focus  groups 
that  would  bring  together  members  of  the  Canadian  intelligence  community  from  a  range  of 
organizations  to  discuss  human  capability  issues  in  intelligence  analysis  and  to  chart  a  way  ahead 
for  the  proposed  survey  study,  and  that  they  were  invited  to  participate  in  that  discussion. 

Twenty-three  subject  matter  experts  from  various  organizations  within  the  intelligence  community 
participated  in  one  of  four  focus  groups.  Participants  represented  a  range  of  roles  including 
intelligence  analysts  ( n  =  6)  managers  (n  =  9),  and  trainers  (n  =  4).  The  remainder  of  participants 
indicated  that  they  held  other  roles  within  the  intelligence  community  that  did  not  fall  into  the 
previous  categories  (n  =  4).  Participants  represented  eight  different  organizations  including  the 
Canadian  Border  Services  Agency  (CBSA),  Canadian  Forces  School  of  Military  Intelligence 
(CFSM1),  Canadian  Security  Intelligence  Service  (CS1S),  Chief  of  Defence  Intelligence  (CD1), 
Integrated  Terrorism  Assessment  Centre  (1TAC),  Privy  Council  Office  (PCO),  Public  Safety 
Canada  (PSC),  and  the  Royal  Canadian  Mounted  Police  (RCMP). 


2.2  Procedure 

Each  focus  group  session  began  with  an  introduction  to  the  research  team,  followed  by  a  briefing 
that  informed  participants  of  the  purposes  of  the  study,  its  relevance  and  potential  benefit  to  the 
intelligence  community  as  a  whole.  Each  member  of  the  focus  group  was  asked  to  provide  a  brief 
introduction  to  the  group,  following  which  the  hosts  clearly  outlined  the  purposes  of  the  session. 
Each  focus  group  was  conducted  over  a  4-hour  period  in  a  roundtable  format. 

This  focus-group  research  was  conducted  in  two  phases,  as  described  fully  below.  The  main 
purpose  of  Phase  1  was  to  identify  capability  challenges  in  intelligence  analysis  within  the  human 
domain  and  to  assess  the  importance  of  these  challenges,  whereas  the  main  purpose  of  Phase  2  was 
to  validate,  expand,  and  elaborate  on  the  challenges  and  other  issues  identified  in  Phase  1 . 

2.2.1  Phase  1 

The  first  two  focus  groups,  comprising  Phase  1 ,  were  convened  in  August,  20 1 1 ,  in  Ottawa. 
Thirteen  subject  matter  experts  participated  in  Phase  1.  Following  the  preliminaries,  the  focus 


Human.vv.vicm 


Human  Capability  Challenges  in  Intelligence  Analysis 


Page  3 


HUMANSKSrEWS 


groups  were  organized  around  three  sessions,  the  first  two  of  which  were  themed  and  focused  on 
identifying  capability  challenges.  The  final  session,  as  we  discuss  below,  involved  revisiting  the 
material  elicited  in  the  themed  sessions.  The  themes  used  in  the  first  two  sessions  were  intended  to 
help  participants  structure  their  observations  and  increase  the  likelihood  that  the  information 
provided  would  also  be  of  use  to  the  research  team. 

The  first  session  was  organized  around  a  “levels  of  organization”  theme.  The  organizers  began  the 
session  by  noting  that  capability  challenges  could  occur  at  various  levels,  including  the  individual, 
group  or  team,  organization,  all  the  way  up  to  the  community.  Before  beginning  the  discussion, 
participants  were  asked  to  individually  rate  the  importance  of  each  of  the  four  levels  of 
organization  on  a  5-point  scale  ranging  from  1  (of  little  or  no  value)  to  5  (extremely  valuable)  with 
3  (quite  valuable)  as  the  midpoint  of  the  scale.  Participants  were  asked  to  begin  with  a  discussion 
of  issues  that  they  regarded  as  important  at  the  individual  level,  and  the  organizers  transitioned  the 
discussion  to  the  team,  organizational,  and  community  levels,  in  turn.  Using  this  framework, 
participants  reflected  on  and  discussed  the  challenges  encountered  within  the  intelligence  domain 
at  each  of  these  levels.  In  practice,  however,  there  was  considerable  “cross-over”  once  the 
discussions  began,  and  the  organizers  did  not  force  participants  to  restrict  their  comments  to  only 
those  congruent  with  the  particular  level  formally  being  discussed.  The  framework  was  meant  to 
facilitate  and  structure  discussion.  However,  it  was  not  meant  to  curtail  it  or  cause  it  to  be  stilted. 

The  second  session  explored  intelligence  analysis  from  a  “functions  of  science”  perspective.  The 
organizers  began  this  session  by  noting  that  applied  behavioural  science  research  can  be  described 
in  terms  of  three  primary  functions  or  activities  including  clearly  documenting  current  tools, 
methods  or  practices;  rigorously  evaluating  the  quality  or  efficacy  of  current  or  proposed  tools, 
methods,  or  practices;  and  developing  new  tools,  methods,  or  practices.  The  organizers  explained 
that  the  aim  of  the  second  session  was  to  explore  how  the  various  challenges  raised  during 
discussion  might  be  best  addressed  through  behavioural  science.  Discussion  started  with  issues  that 
would  require  mainly  a  documenting  function,  and  then  moved  on  to  explore  the  evaluation,  and 
development  functions.  Prior  to  beginning  the  discussion,  participants  rated  the  importance  of  each 
of  the  three  functions  of  science  on  the  same  5-point  scale  used  in  the  first  session. 

By  approaching  the  capability  challenge  question  from  these  two  distinct  perspectives  (level  of 
organization  and  scientific  function),  we  hoped  to  triangulate  the  most  serious  challenges  raised 
with  ways  in  which  a  program  of  applied  behavioural  science  research  might  help  address  some  of 
those  challenges.  Such  information  could  be  useful  in  prioritizing  future  R&D  activities  in  support 
of  the  intelligence  community. 

In  both  of  the  themed  sessions,  the  information  elicited  from  participants  was  recorded  on 
flipcharts  by  a  member  of  the  research  team  at  the  level  specified  by  participants.  Another  research 
team  member  served  as  the  note  taker  and  documented  the  focus  group  discussions  in  detail. 

These  discussions  yielded  a  number  of  issues  and  topics  within  each  of  the  7  categories  (i.e., 
individual,  group,  organization,  and  community  in  Session  1,  and  document,  evaluate,  and  develop 
in  Session  2).  During  the  ensuing  break,  the  research  team  prepared  a  typed  list  of  the  topics  raised 
by  the  participants  organized  by  the  seven  categories  just  noted.  Following  the  break,  in  Session  3, 
participants  were  asked  to  rate  the  potential  value  of  future  research  addressing  each  issue  from 
their  own  perspective  on  the  same  5-point  scale  that  was  used  to  rate  the  value  of  each  level  of 
organization  and  function  of  science  at  the  beginning  of  sessions  one  and  two,  i.e.,  the  scale  ranged 
from  1  (of  little  or  no  value)  to  5  (extremely  valuable)  with  3  (quite  valuable)  as  the  midpoint  of  the 
scale. 
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Finally,  participants  responded  to  eight  additional  questions  about  the  intelligence  process.  These 
questions  addressed  the  relative  importance  of  the  process  and  outcome  evaluations  in  assessing  the 
quality  of  analytic  products;  the  importance  of  having  analytic  standards  and  the  extent  to  which 
these  standards  are  already  in  place  in  participants’  organizations  and  the  community;  the  potential 
value  of  behavioural  science  research  to  improving  analytic  standards;  and  the  relative  importance 
of  description  and  prediction  in  intelligence  reporting.  The  specific  questions  posed  to  the 
participants  and  the  response  options  for  each  of  them  are  presented  in  Annex  A. 

2.2.2  Item  categorization 

Although  the  first  two  focus  groups  identified  many  issues  and  topics  at  the  seven  levels  of 
analysis,  there  was  a  considerable  redundancy  in  the  issues  that  were  identified  by  the  groups.  For 
example,  the  issue  of  career  paths  was  identified  as  being  important  at  multiple  levels  (e.g., 
individual,  group,  organizational).  In  order  to  provide  the  Phase  2  focus  groups  with  the  best 
possible  framework  of  topics  to  discuss  and  rate,  it  was  important  to  identify  the  common, 
underlying  themes  within  the  items  generated  in  the  first  two  focus  groups  and  to  group  them  into 
meaningful  categories.  Following  the  completion  of  the  Phase  1,  the  researchers  sorted  the  issues 
that  emerged  from  Phase  1  and  identified  specific  category  labels  for  the  discrete  themes  within 
each  category.  Overall,  ten  broad  categories  were  identified  (each  containing  specific  issues  and 
topics)  that  subsumed  the  many  diverse  issues  identified  during  the  Phase  1 .  The  item 
categorization  process  is  documented  in  Annex  B,  and  its  outcome  is  shown  in  Table  2. 


2.2.3  Phase  2 

The  third  and  fourth  focus  groups,  which  comprised  Phase  2,  were  convened  in  September,  2011, 
in  Ottawa.  Ten  subject  matter  experts  participated  in  Phase  2.  These  focus  groups  had  a  different 
structure  than  the  earlier  groups.  These  groups  used  the  thematically  sorted  list  generated  in  the 
previous  two  focus  groups  as  a  preliminary  starting  point.  Participants  were  first  asked  to  rate  the 
importance  of  the  29  core  issues  stemming  from  the  Phase  1  discussions. 

In  subsequent  discussions,  this  set  of  29  issues  (categorized  into  10  broad  issues  each  containing  a 
varying  number  of  specific  topics)  served  as  the  starting  points  for  discussions.  In  each  of  the  three 
sessions  (about  40  minutes  each)  a  small  set  of  the  ten  broad  issues  was  presented  on  a  flipchart. 
Participants  were  asked  to  discuss  their  appropriateness  (deleting  or  changing  items  if  necessary), 
to  expand  the  list  if  necessary,  and/or  to  prioritize  the  issues  from  their  unique  perspectives.  In  each 
of  these  three  discussion  sessions,  the  allotted  time  was  used  to  explore  the  listed  challenges,  as 
well  as  to  identify  specific  research  studies  that  might  help  to  address  a  relevant  challenge. 

At  the  end  of  the  session,  additional  discussion  was  then  directed  at  the  content  of  a  community¬ 
wide  survey  for  the  intelligence  community  and  toward  exploring  a  practitioner-researcher 
collaboration  network. 
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Results 


We  present  the  results  of  this  research  in  two  major  sections.  The  first  section  presents  the 
quantitative  assessments  made  by  participants,  and  the  second  section  summarises  the  focus  group 
discussions. 

3.1  Quantitative  evaluations 

3.1.1  Evaluations  of  the  seven  themes  in  the  Phase  1 

Before  the  start  of  the  discussions  in  the  Phase  1,  participants  rated  the  importance  of  each  of  the 
seven  themes  that  were  used  to  structure  the  discussions,  namely,  the  four  levels  of  organization 
(individual,  group,  organization,  and  community)  and  the  three  functions  of  science  (documenting, 
evaluating,  and  developing).  The  ratings  for  most  of  the  themes  were  similar  and  averaged  around 
4  (“very  valuable”)  with  an  exception  of  the  “documenting”  function  of  science,  which  received  a 
somewhat  lower  average  rating  of  3  (“quite  valuable”).  Table  1  reports  the  mean  ratings.  Similar 
ratings  were  not  collected  from  the  Phase  2  participants,  because  the  discussions  in  the  phase  two 
were  structured  not  around  the  themes  but  around  the  categories  of  issues  generated  by  the  Phase  1 
participants. 

Table  1.  Importance  of  research  in  each  of  the  seven  themes,  Phase  1  (N=  13) 


Session 

Theme 

Mean  rating 

Levels  of 
organization 

Individual  level  research 

4.1 

Group  level  research 

3.5 

Organization  level  research 

3.8 

Community  level  research 

3.8 

Functions 
of  science 

Documenting  current  practices 

3.2 

Evaluating  current  practices 

3.8 

Developing  new  practices 

3.9 

3.1.2  Quantitative  evaluations  of  the  categorized  list  of  topics  generated  in  the 
Phase  1 

Several  specific  capability  challenges  were  identified  during  the  Phase  1  discussions,  which  were 
later  categorized  by  the  research  team  into  a  list  presented  in  Table  2.  Participants  in  both  the  Phase 
1  and  Phase  2  rated  the  importance  of  some  or  all  of  these  issues.  At  the  end  of  their  focus  group 
sessions,  Phase  1  participants  evaluated  the  importance  of  each  of  the  capability  challenges  that 
their  group  had  identified  during  the  discussion.  In  contrast,  participants  of  the  Phase  2  evaluated 
the  entire  categorized  set  of  challenges  at  the  beginning  of  their  sessions.  The  differences  in 
methodology,  therefore,  do  not  license  a  simple  averaging  of  the  data  across  the  phases. 
Accordingly,  in  Table  2,  which  reports  the  mean  ratings  of  the  value  of  research  exploring  each  of 
the  specific  topics  in  the  categorized  list  of  issues,  the  mean  ratings  for  Phases  1  and  2  are  reported 
separately.  The  original  list  of  issues  generated  by  the  Phase  1  participants  along  with  their  ratings 
are  included  in  Annex  C,  and  the  categorization  process  is  presented  in  Annex  B. 
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Table  2.  Ratings  of  specific  topics  in  the  categorized  list  of  capability  challenges 


Broad  category 

Specific  topic  or  issue 

Mean  rating 
Phase  1 

Mean  rating 
Phase  2 

1)  Analyst  recruitment, 
motivation  and  career 
path 

1.1  -  Identifying  analyst  selection  criteria  and 
developing  selection  tools; 

3.8 

3.8 

1 .2  -  Identifying  drivers  of  analyst  motivation  for 
recruitment,  retention,  and  career  progression. 

3.7 

3.6 

2)  Analysis  of  tasks  and 
competencies 

2.1  -  Conducting  task  analysis  for  different  roles 
(e.g.,  analyst,  manager,  collector); 

3.6 

3.4 

2.2  -  Identifying  competencies  for  different  roles  and 
tasks  (e.g.,  communication). 

3.6 

3.5 

3)  Performance 

Evaluation 

3.1  -  Defining  expectations  of  analysts' 
performance; 

4.0 

3.8 

3.2  -  Developing  objective  evaluation  methods  of 
process  and  products; 

4.2 

3.5 

3.3  -  Developing  tools  to  support  performance 
evaluation  and  audit  trails. 

3.6 

3.5 

4)  Mentoring  and 

Feedback 

4.1  -  Identifying  effective  methods  for  the  mentoring 
of  analysts; 

3.6 

3.4 

4.2  -  Instilling  openness  and  acceptance  to 
feedback  and  criticism  (and  understanding  the 
precursors  of  the  lack  of  openness  to  feedback). 

3.1 

3.5 

5)  Producer/Consumer 
Relationship 

5.1  -  Clarifying  the  optimal  producer-consumer 
relationship  (and  the  manager’s  role  in  this 
process); 

3.5 

3.9 

5.2  -  Clarifying  the  analyst’s  role  in  understanding 
and  defining  product  requirements; 

4.1 

3.9 

5.3  -  Educating  consumers  about  intelligence; 

4.2 

4.1 

5.4  -  Identifying  how  products  impact  decision¬ 
makers. 

3.8 

3.7 

6)  Analyst-Collector 
Understanding  and 

Terms  of  Reference 

3.9 

3.6 

7)  Collegial  Collaboration 

7.1  -  Determining  effective  structures  for  analytic 
teams; 

4.2 

3.3 

7.2  -  Instilling  a  collaborative  spirit  and  promoting 
informal  networks. 

3.4 

3.4 

8)  Communication  of 
Intelligence 

8.1  -  Understanding  and  improving  the 
communication  of  uncertainty 

3.6 

3.7 

8.2  -  Utilizing  information  visualization  techniques 
with  consumers 

4.0 

3.7 

8.3  -  Impact  of  existing  formats  and  varying 
communication  of  reports 

3.6 

3.1 

9)  Tools,  Techniques  and 
Practices  (TTPs) 

9.1  -  Documenting  and  evaluating  current  TTPs  and 
developing  new  TTPs. 

3.8 

3.8 

10)  Intelligence 

Community 

10.1  -  Understanding  constraints  and  barriers  to 
knowledge  use  imposed  by  organizational  structure 

3.7 

3.7 

10.2  -  Better  coordination,  collaboration,  and 
information  sharing  among  agencies: 

3.8 

4.5 
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Broad  category 

Specific  topic  or  issue 

Mean  rating 
Phase  1 

Mean  rating 
Phase  2 

10.2.1  -  Facilitating  collaboration  within  the 
community; 

4.0 

4.1 

10.2.2  -  Developing  a  resource  sharing  mechanism; 

4.0 

3.5 

10.2.3  -  Developing  standards  for  different 
information  sources. 

3.6 

3.2 

10.3  -  Cross-organizational  review  of  doctrine,  tools, 
techniques  and  practices. 

3.5 

3.3 

10.4  -  Professionalization  of  intelligence  analysis: 

4.1 

3.8 

10.4.1  -  Developing  common  standards; 

3.9 

3.2 

10.4.2  -  Developing  community-wide  career  path  for 
analysts 

3.5 

3.7 

Table  2  shows  that  research  on  all  of  the  issues  and  topics  identified  in  Phase  1  was  rated  to  be 
quite  valuable  by  participants  in  both  phases.  All  of  the  mean  ratings  are  on  or  above  the  midpoint 
of  the  scale.  Issues  related  to  educating  consumers  about  intelligence,  clarifying  the  analyst’s  role 
in  understanding  requirements,  and  professionalization  of  intelligence  analysis  were  among  the 
highly  rated  categories  in  both  phases.  Developing  information  sources  standards  within  the 
community,  cross-organizational  review  of  doctrine  and  TTPs,  and  instilling  collaborative  spirit 
and  promoting  informal  networks  were  among  the  lower  rated  categories  in  both  phases. 

To  further  explore  issues  of  interest,  focus  group  participants  in  the  Phase  1  were  also  asked  several 
questions  at  the  completion  of  the  sessions  (see  Annex  A  for  the  list  of  questions  with 
corresponding  response  options).  These  results  are  reported  below. 


3.1.3  Process-outcome  distinction  in  evaluating  quality  of  products  and  analysts’ 
performance 

Intelligence  analysis  can  be  evaluated  in  terms  of  the  products  or  outcomes  produced  during  the 
analysis,  or  in  terms  of  the  processes  used  to  undertake  analysis.  As  such,  it  is  theoretically  possible 
to  produce  a  good  product  using  a  flawed  process  (or  vice  versa).  A  majority  (62%)  of  participants 
reported  being  familiar  with  the  process-outcome  distinction,  while  the  remaining  38%  were  not 
familiar  with  it. 

Participants  were  also  asked  to  rate  the  relative  importance  of  process  and  outcome  evaluations  in 
judging  intelligence  activities.  Process  and  outcome  evaluations  were  rated  as  more  or  less  evenly 
balanced  in  terms  of  their  importance  (average  rating  was  3.1,  where  the  rating  of  3  corresponds  to 
“Equally  balanced”).  Figure  1  shows  the  distribution  of  the  responses. 
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Figure  1.  Distribution  of  participants’  responses  regarding  the  relative  importance  of  process  and 
outcome  evaluations  in  judging  intelligence  activities 


Relative  importance  of  process  and  outcome 
evaluations 


1  -  All  2  -  Mostly  3  -  Equally  4  -  Mostly  5  -  All 
process  process,  a  bit  balanced  outcome,  a  outcome 
of  outcome  bit  of  process 


3.1.4  Analytic  standards 

Figure  2  shows  the  mean  ratings  of  the  importance  of  having  analytic  standards  in  place  at 
organizational  and  community-wide  levels,  and  Figure  3  shows  the  mean  ratings  of  the  extent  to 
which  analytic  standards  are  already  in  place. 

Figure  2.  Mean  ratings  of  the  importance  of  having  analytic  standards  at  organizational  and 
community-wide  levels 

Importance  of  analytic  standards 


■  In  your  organization  ■  In  the  community 
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Figure  3.  Mean  ratings  of  the  extent  to  which  analytic  standards  are  already  in  place 

The  extent  to  which  the  standards 
are  already  in  place 

all  the  way 
there 

about  % 

about  V2  of 
the  way 
there 

about  % 

not  at  all 

B  In  your  organization  ■  In  the  community 

Although  having  standards  was  rated  as  quite  valuable  within  both  one’s  own  organization  and  in 
the  community,  more  generally,  participants  also  rated  this  as  a  current  area  of  concern,  on  average, 
reporting  that  their  own  organizations  were  only  %  of  the  way  toward  having  these  standards  in 
place.  As  Figure  3,  shows,  participants,  on  average,  also  thought  that  the  community  had  an  even 
further  way  to  go  in  implementing  standards. 

Given  that  one  of  the  goals  of  the  focus  groups  was  to  determine  how  behavioural  science  research 
might  facilitate  the  intelligence  community,  participants  were  also  asked  about  how  helpful 
behavioural  science  could  be  in  facilitating  standards.  Behavioural  science  was  seen  as  having 
some  potential  role  in  promoting  better  standards  within  the  community,  receiving  the  mean  rating 
of  2.9,  which  corresponded  to  “quite  valuable”  on  the  relevant  scale. 


3.1.5  Description  versus  Prediction  in  intelligence  analysis 

Lastly,  participants  were  also  asked  to  rate  the  importance  of  description  and  prediction  in 
intelligence  analysis.  The  question  was  posed  differently  to  the  two  focus  groups;  however,  the 
results  were  obtained  from  the  two  groups  were  very  similar.  In  the  first  focus  group,  participants 
evaluated  the  relative  importance  of  description  or  prediction  on  a  single  scale  that  ranged  from  1 
(description  is  important,  prediction  is  not)  to  5  (prediction  is  important,  description  is  not).  The 
mean  rating  in  the  first  group  was  3.3,  which  roughly  corresponds  to  the  midpoint  of  the  scale 
indicating  that  description  and  prediction  are  equally  important.  In  the  second  focus  group, 
participants  were  asked  to  evaluate  the  importance  of  description  and  prediction  separately  and 
both  aspects  received  the  same  mean  rating  of  3.9,  which  corresponds  to  the  “very  valuable”  level 
on  the  scale.  Thus,  prediction  and  description  were  seen  being  more  or  less  of  equal  importance 
regardless  of  the  response  mode  used  by  the  researchers  to  elicit  participants’  views. 
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3.2  Summary  of  the  discussions 

In  all  four  focus  groups,  the  majority  of  the  session  was  devoted  to  a  discussion  of  the  human 
capability  challenges  in  the  intelligence  community.  The  following  section  summarises  these 
discussions  and  presents  them  organized  by  topics  listed  in  Table  2. 


3.2.1  Analyst  Recruitment,  Motivation,  and  Career  Path  (Recruitment  and 
Retention) 

The  recruitment  and  retention  of  intelligence  analysts  were  identified  as  challenges  facing  the 
intelligence  community  in  all  four  focus  groups.  One  of  the  main  human  resource  challenges  that 
participants  identified  was  the  lack  of  validated  selection  tools,  including  the  lack  of  guidelines  on 
the  appropriate  selection  process  for  intelligence  professionals.  Job  descriptions  and  the  required 
qualifications  and  skills  for  these  positions  remain  very  general  for  many  intelligence  positions 
within  the  community.  Although  this  does  not  allow  for  an  effective  screening  process,  on  the  other 
hand,  this  allows  for  a  sufficient  flexibility  in  balancing  the  needs  of  the  organization  with  the 
resources  that  are  most  readily  available.  Participants  remarked,  however  that  organizations  vary  in 
their  ability  to  properly  evaluate  prospective  candidates.  Some  participants  also  noted  that  the 
quality  of  the  available  recruit  pools  is  also  sometimes  lacking,  and  that  the  selection  process  could 
make  use  of  relevant  cognitive  testing,  such  as  testing  for  analytic  ability  and  critical  thinking 
skills,  to  screen  candidates. 

Following  selection,  the  next  major  challenge  for  the  community  is  personnel  retention. 

Participants  discussed  what  motivates  an  analyst  to  be  (and  to  continue  to  be  an  analyst),  as  this 
seems  likely  to  be  a  key  predictor  of  retention.  Some  participants  said  that  one  important  motivator 
for  analysts  is  to  have  decision-making  responsibility  over  choosing  the  type  of  analysis  for  a 
problem..  Some  participants  also  suggested  that  senior  analysts  can  be  motivated  by  opportunities 
to  get  involved  in  higher-level  activities  and  by  opportunities  to  give  back  to  the  profession, 
including  by  contributing  to  the  professional  development  of  junior  colleagues. 

Varying  the  scope  and  variety  of  the  work  that  intelligence  analysts  do  may  help  promote  better 
retention  as  well.  A  common  concern  in  all  four  focus  groups  was  the  limited  career  progression 
opportunities  for  intelligence  analysts  in  their  organizations  and  in  the  community,  which  is  often 
constrained  by  their  size  and  mandate.  The  number  of  senior  analyst  positions  available  is  small 
and  the  conditions  for  promotion  to  those  positions  are  not  clearly  defined.  Therefore,  competent 
analysts,  who  seek  further  professional  development  and  career  challenges,  are  likely  to  look  for 
advancement  opportunities  elsewhere.  The  resulting  turnover  can  have  negative  impact  on  the 
accumulation  of  knowledge  within  organizations.  The  fact  that  the  Canadian  intelligence 
community  is  relatively  small  makes  the  career  path  issue  even  more  challenging. 

The  work  of  intelligence  analysts  has  a  number  of  intrinsic  stressors  that  could  also  influence  an 
organization’s  ability  to  retain  analysts.  Some  of  the  most  frequently  mentioned  stressors  were  the 
“tyranny  of  production”  and  “speaking  truth  to  power.”  The  performance  of  analysts  is  often 
measured  by  the  number  of  assessments  that  they  complete.  Production  typically  occurs  under 
intense  time  pressure  and  ensuring  acceptable  quality  puts  considerable  stress  on  analysts,  which  is 
termed  the  “tyranny  of  production.”  The  pressure  associated  with  “speaking  truth  to  power”  arises 
when  issues  that  analysts  identify  are  not  popular  with  their  consumers,  and,  as  a  result,  analysts 
have  to  manage  the  need  to  provide  accurate  information  to  their  manager  and  consumers  while 
protecting  their  own  career  interests. 
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In  addition,  participants  noted  that  compensation  levels  for  analysts  are  below  that  of  industry  and 
often  they  do  not  match  the  increasing  expectations  of  current  recruits.  Some  participants  suggested 
that  preventing  the  loss  of  hard  won  expertise  and  ensuring  increased  long-term  retention  may 
require  further  research  into  the  motivating  factors  for  intelligence  professionals  and  the 
development  of  better  career  paths  for  analysts. 

3.2.2  Tasks  and  Competencies 

One  of  the  core  questions  that  remains  to  be  answered  according  to  some  participants  is  “What 
skills  and  abilities  do  you  need  to  be  an  analyst?”  Participants  discussed  a  number  of  skills  that 
analysts  require,  such  as  having  a  high  level  of  interest  and  motivation,  passion,  critical  thinking 
skills ,  decision-making  skills,  attention  to  detail,  openness  to  criticism,  general  open-mindedness, 
clear  and  concise  expression  of  one ’s  thoughts,  ability  to  perform  under  pressure,  and  staying  open 
to  the  facts.  Some  participants  noted  that  although  expertise  is  regarded  as  a  good  thing,  it  may  also 
be  associated  with  its  own  challenges  such  as  developing  tunnel  vision,  disregard  for  the  input  or 
feedback  of  others,  and  isolation  from  others.  Trying  new  opportunities  within  the  organization  was 
identified  as  one  way  to  counter  these  challenges. 

Although  personality  and  individual  characteristics  can  certainly  impact  the  performance  of 
analysts,  participants  also  argued  that  the  real  skills  required  for  analysis  are  typically  developed  by 
the  experience  of  being  an  analyst.  Participants  noted  that  advanced  academic  degrees  do  not 
always  give  analysts  an  advantage,  as  such  analysts  may  tend  to  overcomplicate  things,  have 
difficulty  in  adjusting  their  academic  writing  style  to  the  evidence-based  succinct  intelligence 
reporting  style,  and  could  also  be  more  rigid  in  accepting  alternative  perspectives  and  criticism. 

Participants  recognised  that  there  is  a  gap  in  understanding  the  various  tasks  and  competencies  for 
different  roles  (e.g.,  analyst,  manager,  collector,  etc.)  within  the  intelligence  community.  They  also 
argued  that  cognitive  task  analysis  could  help  advancing  this  understanding  by  documenting  the 
demands  on  collectors,  analysts,  and  managers.  Understanding  these  demands  together  with 
identifying  the  skills  that  feed  into  the  core  competencies  would  be  a  helpful  contribution  for  future 
research  to  make. 

3.2.3  Performance  Evaluation 

The  discussions  also  elaborated  on  the  enduring  challenges  associated  with  evaluating  performance 
within  the  intelligence  community.  Among  these  challenges  were  the  lack  of  clear  set  of 
expectations  and  objective  indicators  of  analysts’  performance;  the  lack  of  information  on  the 
usage  and  feedback  from  the  consumers  on  the  quality  of  the  reports;  and  the  general  lack  of 
appropriate  performance  evaluation  tools.  In  addition,  some  participants  noted  that  performance 
evaluation  is  not  necessarily  linked  to  salary  increases,  which  could  in  turn  create  the  perception  of 
it  as  a  token  process. 

Sound  performance  evaluation  practice  should  be  based  on  clear  performance  expectations  that  are 
linked  to  objective  performance  indicators.  Participants  discussed  a  number  of  relevant  indicators 
that  could  be  considered,  such  as  the  predictive  accuracy  of  an  intelligence  product,  the  actual 
impact  of  the  product,  its  timeliness,  consumer  satisfaction,  and  its  creative  aspects.  However, 
information  on  many  of  these  indicators  is  generally  unavailable  and,  even  if  it  were  available,  it  is 
often  very  difficult  to  quantify.  Therefore,  developing  methodology  for  gathering  relevant 
information  and  for  quantifying  the  possible  performance  indicators  was  identified  as  an  important 
effort  for  the  community. 
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Consequently,  participants  discussed  the  value  of  finding  a  way  to  document  the  actual  usage  of 
intelligence  products  that  would  assess  whether  the  information  produced  reaches  the  target 
audiences,  how  often  consumers  actually  use  the  products,  and  what  aspects  of  the  report  they  give 
attention  to  and  for  how  long.  Similarly,  some  participants  noted  that  gathering  feedback  from 
consumers  and  understanding  how  well  the  products  suit  their  needs  would  also  be  a  critical 
indicator  of  the  products’  impact.  Although  the  access  to  the  consumers  (and  therefore,  availability 
of  their  feedback)  varies  among  the  organizations,  the  prevalent  opinion  in  the  focus  groups  was 
that  feedback  from  the  consumers  was  largely  unavailable. 

Participants  noted  that  intelligence  analysis  can  be  performed  well  and,  yet,  not  be  valuable  to  the 
client.  Therefore,  evaluation  should  encompass  assessment  of  both  the  process,  by  which  an 
intelligence  product  was  created,  and  the  intelligence  product.  This  argument  echoes  the  results  of 
the  quantitative  evaluations  of  the  relative  importance  of  product  and  process  evaluation  collected 
from  the  Phase  1  participants. 

Focus  group  discussions  emphasized  that,  in  order  to  evaluate  the  intelligence  process,  intelligence 
organizations  need  to  develop  defendable,  reviewable  processes  and  procedures  for  audit  trails. 
Analysts  need  to  be  able  to  show  the  tracking  of  the  information  that  they  used  to  arrive  at  their 
conclusions,  which  could  serve  as  a  basis  for  an  objective  process  evaluation.  Audit  trails  would 
show  how  the  information  supports  the  main  findings  and  conclusions  of  the  analysis,  which  would 
also  illustrate  the  decision-making  processes,  and  such  trails  would  also  show  how  the  intelligence 
analyst  reached  their  conclusion.  Participants  argued  that  audit  trails  could  promote  greater 
acceptance  by  the  consumers  of  conclusions  drawn  in  intelligence  products. 

Although  the  majority  of  the  discussion  in  the  focus  groups  centered  on  evaluation  of  an  individual 
analyst  or  a  single  product,  there  was  also  discussion  about  the  potential  value  of  being  able  to 
evaluate  the  performance  of  units  or  organizations  within  the  intelligence  community.  Participants 
argued  that  it  would  be  ideal  to  audit  the  performance  of  an  intelligence  unit  as  a  broader  form  of 
organizational  performance. 

Overall,  the  area  of  individual  and  organizational  performance  evaluation,  then,  stands  as  a 
particularly  important  area  of  future  research  that  could  be  beneficial  to  the  intelligence 
community. 

3.2.4  Mentoring  and  Feedback 

The  transfer  of  knowledge  from  experienced  analysts  to  more  junior  personnel  was  identified  as 
another  key  challenge  in  the  community.  Some  participants  suggested  that  this  problem  might  be 
mitigated  through  better  mentoring  and  feedback.  Participants  generally  expressed  strong  support 
for  the  concept  of  mentoring  and  indicated  that  their  organizations  are  actively  working  to  promote 
mentorship.  For  example,  the  addition  of  mentoring  onto  the  balanced  scorecard  template,  which  is 
used  in  various  government  departments,  serves  as  recognition  of  its  positive  impact  within  the 
community. 

Flowever,  some  participants  also  noted  that  mentoring  can  be  taxing  on  mentors,  and  that  the  best 
mentors  (e.g.,  those  with  high  levels  of  expertise)  are  often  least  able  to  afford  the  time  to  mentor. 
Therefore,  meaningful  organizational  support  for  mentoring  is  required,  such  as  the  alleviation  of 
work  stresses  and/or  improved  reward  structures  that  provide  incentives  for  experienced  personnel 
to  mentor.  Participants  indicated  that,  currently,  these  reward  structures  were  lacking.  Participants 
suggested  that  empirical  evidence  of  the  positive  impact  of  mentoring  could  facilitate  the 
implementation  of  these  organizational  incentives. 
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The  ability  of  intelligence  personnel  to  provide  constructive  feedback  and,  likewise,  their  ability  to 
receive  feedback  and  criticism  was  identified  as  an  important  challenge.  Participants  noted  that 
intelligence  analysts  often  lack  both  of  these  skills.  Developing  in  intelligence  personnel  greater 
openness  to  criticism  and  helping  them  to  learn  to  balance  the  provision  of  positive  and  negative 
feedback  more  adeptly  were  seen  as  important  issues  that  required  attention. 

3.2.5  Producer-Consumer  Relationship 

Discussions  about  the  producer-consumer  relationship  were  prominent  within  all  of  the  focus 
groups  and  highlighted  a  number  of  issues.  A  range  of  organizations  represented  in  the  focus 
groups  also  provided  a  variety  of  perspectives  on  these  issues  and  their  prominence.  Participants 
agreed  that  the  nature  of  the  producer-consumer  relationship  (actual  and  optimal)  may  change  as 
one  moves  back  and  forth  between  tactical  and  strategic  levels.  This  suggests  that  although  a  higher 
level  of  clarity  concerning  the  ideal  producer-consumer  relationship  would  be  helpful,  a  clear 
understanding  of  the  many  different  types  of  producer-consumer  relationships  within  the 
community  is  an  important  prerequisite.  The  organizations  represented  in  this  study  produce 
intelligence  for  many  different  types  of  consumers.  Some  organizations  serve  a  long  line  of  users, 
and  this  has  an  impact  on  analysts’  awareness  of  who  their  consumers  are.  Thus,  for  some 
organizations,  identifying  their  consumers  and  receiving  feedback  from  them  is  a  challenge. 

The  discussions  revealed  that  one  of  the  key  issues  with  which  the  community  is  struggling  is  lack 
of  clear  lines  of  communication  between  intelligence  producers  and  consumers.  This  issue 
manifests  itself  in  the  lack  of  clear,  specific  and  timely  product  requirements,  and  later,  in  the  lack 
of  feedback  on  the  final  product  from  the  consumers.  The  disconnection  between  intelligence 
producers  and  consumers  is  deepened  by  their  cultural  differences  and  the  common  “arms-length” 
relationship  between  analysts  and  consumers  that  is  intended  to  protect  the  analyst  from  influence 
that  might  impede  their  ability  to  deliver  an  unbiased  assessment.  Although  this  type  of  relationship 
may  protect  the  analysts,  this  distance  often  results  in  the  lack  of  mutual  understanding. 

Participants  commented  that  producers  often  do  not  understand  how  consumers  actually  use  their 
products  and  how  their  products  impact  decision-makers.  The  consumers  often  do  not  clearly 
understand  the  intelligence  process,  its  capabilities  and  limitations. 

Participants  argued  that  many  intelligence  consumers  do  not  have  an  accurate  understanding  of 
how  to  formulate  their  questions.  Accordingly,  there  is  a  need  to  accurately  define  consumer 
requirements  and  determining  what  role  analysts  or  managers  should  play  in  this  process.  In 
addition,  consumers  might  not  fully  appreciate  the  professional  process  required  to  conduct 
intelligence  properly,  and  what  information  intelligence  can  and  cannot  provide.  This  lack  of 
knowledge  on  the  part  of  consumers  can  affect  the  entire  production  process  as  well  as  consumers’ 
acceptance  and  usage  of  the  final  product.  Consequently,  some  of  the  important  challenges  within 
the  intelligence  community  that  participants  identified  were  the  need  for: 

i)  educating  consumers  about  intelligence; 

ii)  formal  feedback  mechanisms  on  the  quality  and  usefulness  of  intelligence  products; 

iii)  developing  tools  that  will  allow  tracking  the  actual  usage  of  products; 

iv)  developing  tools  to  help  understand  the  full  impact  of  an  intelligence  product  on 
decision  makers. 

The  formal  feedback  and  the  actual  product  usage  statistics  would  help  ensuring  products’ 
relevance  to  the  consumers  and  could  also  be  integrated  into  the  performance  evaluation  metrics. 
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Another  issue  related  to  producer-consumer  relationship  discussed  was  the  politicization  of 
intelligence  tasking.  Participants  noted  that  it  was  impossible  to  totally  separate  intelligence 
analysis  from  the  political  domain,  and  that  the  media  coverage  plays  a  particularly  important  role 
in  influencing  political  responses  to  issues  and  subsequent  intelligence  tasking.  This  is  problematic, 
they  argued,  because  it  often  detracts  attention  from  potentially  more  serious  (but  less  prominent) 
threats.  It  also  might  result  in  duplication  of  efforts  among  different  organizations  that  are  being 
tasked  to  prepare  assessments  of  the  same  issue. 

3.2.6  Analyst-Collector  Understanding  and  Terms  of  Reference 

Some  participants  identified  the  relationship  between  the  analyst  and  collector  as  an  important  one 
with  many  interdependencies.  Participants  argued  that  the  roles  and  responsibilities  of  the  analyst 
and  collector  need  to  be  clarified,  and  that  analysts  needed  to  be  involved  in  (or  at  least  well  aware 
of)  the  collection  process.  Participants  suggested  that  this  relationship  would  differ  at  the  strategic 
versus  the  tactical  level.  For  example,  in  a  strategic  situation,  it  might  be  impossible  to  be  in  touch 
with  an  active  collector  or  to  probe  for  particular  information.  At  the  tactical  level,  however,  this 
relationship  might  be  less  constrained.  Whatever  their  relationship,  focus  groups  indicated  that 
analysts  and  collectors  need  a  dialogue  and  to  establish  their  terms  of  reference  in  order  to  work 
together  more  effectively.  This  is  a  current  gap  within  the  intelligence  community. 

3.2.7  Collegial  Collaboration 

Collegial  collaboration  is  an  important  challenge  for  the  intelligence  community,  because  as  one 
participant  noted  “it  takes  a  network  to  compete  against  a  network.”  There  are  two  aspects  of 
collegial  collaboration  that  were  discussed  by  the  groups:  the  formal  aspect  of  collegial 
collaboration  in  the  context  of  analytic  teams;  and  the  informal  aspect  of  collegial  collaboration  in 
the  context  of  promoting  informal  networks  among  analysts. 

On  a  formal  level,  some  participants  argued  that  there  should  be  more  fluidity  in  the  structures  used 
to  do  intelligence  analysis  at  the  organizational  and  team  level.  Participants  argued  that  research 
exploring  the  impact  of  varying  team  structures  on  the  performance  of  teams  and  their  members 
would  be  helpful. 

Because  Canada  has  a  relatively  small  intelligence  community,  participants  argued,  informal 
collegial  networks  are  highly  instrumental  in  overcoming  one  of  the  serious  problems  -  the  lack  of 
information  sharing  at  the  organizational  and  community  levels.  However,  this  collaboration  is 
often  interpersonal,  ad  hoc  in  nature,  and  might  also  be  negatively  influenced  by  operational 
requirements  such  as  time  pressure  and  a  lack  of  a  mechanism  for  easy  sharing  of  information 
(discussed  in  more  detail  within  Section  3.2.10). 

Given  the  facilitative  role  of  informal  networks,  one  of  the  challenges  for  the  community  according 
to  participants  is  to  find  ways  to  provide  intelligence  personnel  with  opportunities  to  build  informal 
networks  and  to  provide  an  infrastructure  that  will  promote  these  relationships. 


3.2.8  Communication  of  intelligence 

The  need  to  ensure  the  highest  possible  level  of  communication  with  consumers  was  another 
important  theme  emerging  from  the  focus  groups.  Effective  communication  should  be  clear, 
simple,  concise,  and  geared  to  the  audience.  Participants  observed  that  communication  of 
uncertainty  is  an  important  aspect  of  intelligence  reporting  that  currently  relies  on  verbal 
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expressions  of  chance  and  risk.  Verbal  expressions  inherently  allow  for  a  certain  degree  of 
variability  in  their  interpretation,  which  can  undermine  clarity  of  communication.  Participants 
noted  that  one  of  the  community’s  current  challenges  is  the  lack  of  a  community-wide  standard  for 
communicating  chance  and  risk,  and  that  there  is  a  need  for  more  standardization  in  the  language 
used  to  communicate  uncertainty. 

Some  discussion  also  centred  on  structural  tools  or  processes  that  might  facilitate  better 
communication  in  written  reports,  such  as  an  up  front  “bottom-line  assessment”  or  the  “key 
judgements”  section.  Different  presentation  formats,  including  the  use  of  visualisation  techniques 
in  the  reports  were  also  identified  as  factors  that  could  facilitate  communication  of  intelligence. 
Participants  noted  that  the  intelligence  community  lags  substantially  behind  the  private  sector  in  its 
use  of  information  visualization  techniques  and  tools  and  called  for  further  research  and 
development  of  information  presentation  and  management  tools  for  the  community. 

Participants  argued  that  there  are  many  different  ways  in  which  intelligence  could  be  presented  to 
consumers,  but  that  there  is  no  single  best  format.  Different  consumers  may  have  different 
information  needs  and  information  format  preferences.  Therefore,  it  is  critical  for  analysts  to  know 
their  audience  and  to  gear  communication  and  its  presentation  to  their  consumers’  needs.  Research 
exploring  alternative  formats  for  reports  and  other  information  visualization  techniques  could  help 
to  promote  better  communication  with  consumers. 

3.2.9  Tools,  Techniques,  and  Practices  (TTP) 

Although  the  analytic  process  seems  particularly  amenable  to  being  supported  by  technology, 
participants  argued  that  sophisticated  resources  are  not  uniformly  available  to  the  intelligence 
community  and  that  their  use  might  be  prevented  by  the  tight  timelines  for  completing  assessments. 
Some  participants  argued  that  analysts  need  tools  and  techniques  to  help  manage  the  sheer  volume 
of  available  information  and  that  the  community  currently  lacks  the  technology  that  could  assist  the 
information  filtering  process  calling  for  the  development  of  new  tools  that  could  meaningfully 
assist  with  that  process. 

It  is  not  clear  which  of  the  currently  available  tools  are  actually  being  used  by  different 
organizations  and  how  effective  they  are.  Therefore,  cataloguing  tools  that  are  being  used  in  the 
field  and  identifying  personnel  who  have  expertise  in  each  of  them  was  seen  as  a  useful  activity. 
Participants  also  emphasised  the  need  for  empirical  assessment  of  the  effectiveness  of  current 
methodologies  available  or  used  in  intelligence  analysis. 

In  addition,  participants  noted  that  a  research  capability  that  could  simultaneously  help  intelligence 
personnel  to  understand  how  to  define  their  technology  requirements,  paired  with  an  evaluation 
function  that  could  assess  usability,  performance  and  functionality  of  technology  products  would 
be  helpful. 

Another  issue  that  emerged  from  the  discussions  was  the  need  for  documenting  the  methodological 
approaches  and  techniques  that  intelligence  analysts  use  and  for  comparing  them  to  the  prescribed 
methods  (e.g.,  doctrine  and  standards).  In  addition  to  comparing  the  doctrine  and  practice, 
participants  suggested  that  examining  the  congruence  between  the  doctrine  and  human  capability 
would  also  be  a  useful  exercise.  These  activities  would  allow  identifying  and  addressing  gaps 
between  the  doctrine  and  practice. 
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3.2.10  Intelligence  Community 

Within  all  four  focus  groups,  much  discussion  focused  on  the  challenges  facing  the  broader 
intelligence  community. 

Information  sharing  and  coordination 

One  such  challenge  is  the  lack  of  efficient  information  sharing  among  agencies.  Participants 
highlighted  the  need  to  better  understand  existing  barriers  to  information  and  knowledge  sharing. 
Some  of  the  factors  that  contribute  to  the  current  situation  discussed  were:  i)  the  lack  of  trust 
among  organizations,  which  leads  to  rivalries  and  protectionism  of  the  information  that 
organizations  have  acquired;  ii)  organizational  structure  and  bureaucratic  constraints,  such  as 
policy  barriers,  legislative  issues,  and  varying  levels  of  security  clearance  among  agencies. 

According  to  participants,  developing  common  standards  and  guidance  for  evaluating  the  quality  of 
information  sources  and  determining  information  classification  level  would  facilitate  information 
sharing  among  agencies.  In  addition,  participants  noted  that  research  that  documents  the  available 
resources  (technical  or  non-technical)  within  each  organization  and  that  provides  information  about 
how  to  access  these  resources  would  be  particularly  helpful. 

Support  requests  that  specific  organizations  receive  from  consumers  are  also  often  not  shared  with 
other  organizations  and  potentially  several  agencies  might  be  tasked  to  produce  intelligence  on  the 
same  “hot”  topic.  The  Canadian  intelligence  community,  having  a  relatively  small  size,  is 
particularly  prone  to  the  problem  of  overextending  its  resources,  and  that  unclear  or  overlapping 
mandates  among  organizations  can  create  unnecessary  redundancy.  Participants  noted  that  having 
more  clear  delineation  of  roles  and  responsibilities,  promoting  more  of  a  cross  community  view, 
endorsing  joint  projects  within  the  community  as  well  as  developing  better  information  sharing 
mechanisms  could  facilitate  a  higher  level  of  coordination.  One  suggestion  made  during  the  focus 
group  discussions  was  to  create  a  central  repository,  or  a  government  “dashboard”  that  would  allow 
a  quick  overview  of  ongoing  efforts  within  other  parts  of  the  community.  Participants  argued  that 
adopting  a  truly  cross-organizational  focus  might  require  having  the  diverse  intelligence 
organizations  in  Canada  led  by  a  strong  focal  point  (e.g.,  a  national  Director). 

Another  challenge  within  the  community  is  a  lack  of  clear  understanding  of  the  commonalities  and 
differences  in  doctrine,  practice,  process,  and  product  among  the  organizations.  Research  working 
to  promulgate  a  cross-organizational  review  of  the  many  different  players  within  the  community 
would  be  of  benefit.  In  addition,  participants  indicated  that  creating  a  community-wide  portal  that 
would  allow  sharing  doctrine  and  soliciting  community  feedback  and  engagement  would  be  of 
benefit. 

Professionalization 

A  good  deal  of  discussion  focused  on  the  need  for  a  professionalization  of  the  intelligence 
community  that  promotes  best  practices  within  the  community.  Professionalization  was  expressed 
as  a  need  for  the  community  to  reflect  on  the  current  state  of  affairs  within  the  community  and  to 
imagine  the  intelligence  landscape  of  the  future.  For  example,  participants  noted  that  one  of  the 
gaps  within  the  community  is  no  person  or  organization  is  currently  formally  assigned  to  consider 
the  future  development  of  intelligence  processes  and  product.  Similarly,  participants  noted  the  need 
for  a  community-wide  exploration  of  the  question  “what  does  it  mean  to  be  an  analyst?” 

Participants  suggested  that  professionalization  of  the  community  could  be  promoted  through  the 
development  of  common  standards  for  intelligence  products  and  training  of  the  personnel. 
Participants  suggested  that  the  development  of  common  product  templates  and  “reader  guides”  that 
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are  already  implemented  in  some  organizations  would  help  consumers  to  understand  the  reports.  In 
addition,  such  standards  would  also  facilitate  consumer  education  about  intelligence  products. 

Participants  agreed  that  in  addition  to  the  intelligence  training  organized  by  the  Privy  Council 
Office,  more  formalized  standardization  efforts  are  needed  within  the  intelligence  training  system. 
The  development  of  a  list  of  standardized  courses  that  all  personnel  within  the  community  should 
take  was  seen  as  beneficial.  In  addition,  participants  noted  that  intelligence  training  should  also 
place  emphasis  on  laying  out  the  current  intelligence  landscape,  providing  information  about  other 
organizations  performing  intelligence  analysis  and  their  mandates. 

A  common  theme  throughout  the  focus  groups  was  that  although  seeking  additional  standardization 
and  consistency  of  approaches  within  the  community  is  important,  it  is  also  important  to  recognize 
the  uniqueness  of  the  roles  played  by  (and  the  demands  on)  the  many  members  of  the  community. 
Participants  strongly  suggested  that  a  “one  size  fits  all”  approach  will  not  work  within  the 
community. 

As  noted  in  earlier  sections  of  this  report,  inadequate  opportunity  for  career  progression  is  a 
challenge  for  analysts  within  the  intelligence  community.  In  addition,  there  is  a  lack  of  consistency 
in  analyst  job  categories  among  organizations,  which  results  in  seemingly  unjustified  discrepancies 
in  salary  and  governing  collective  agreements  for  analysts  that  perform  similar  functions  in 
different  organizations. 

Discussing  the  limited  career  progression  for  analysts,  participants  emphasized  the  importance  of 
developing  community-wide  career  paths  and  secondment  opportunities  for  analysts  as  a  way  to 
provide  analysts  with  greater  development  opportunities  and  to  promote  retention.  Participants 
observed  that  these  initiatives  would  require  interagency  coordination.  Although,  secondment 
program  appears  to  be  beneficial,  participants  suggested  that  further  research  into  the  actual  impact 
of  secondment  on  organizations  and  personnel  woidd  be  highly  valuable.  Such  research  could  be 
useful  in  guiding  the  education  and  implementation  of  the  secondment  program. 


3.3  Summary  of  Results 

The  four  focus  groups  identified  and  discussed  a  wide  range  of  current  issues  and  challenges  within 
the  intelligence  community  that  could  be  supported  by  future  research.  Conducting  the  focus 
groups  in  two  different  phases  allowed  to  both  freely  identify  issues  of  concerns  and  to  reflect 
critically  and  elaborate  on  the  identified  issues.  The  Phase  2  participants  indicated  that  the 
categories  compiled  from  the  Phase  1  were  appropriate  and  in  line  with  their  experience.  The 
relative  high  importance  ratings  given  by  the  Phase  2  participants  to  all  of  the  items  served  as  an 
additional  validation  for  the  set  of  issues  compiled  from  the  Phase  1 . 

Quantitative  ratings  of  importance  of  different  issues  that  were  gathered  from  participants  of  both 
phases  were  intended  to  provide  a  means  of  differentiating  and  prioritising  the  various  capability 
challenges  identified.  However,  all  issues  received  relatively  high  mean  importance  ratings  in  both 
phases,  and,  therefore  these  ratings  cannot  serve  as  a  meaningful  basis  for  prioritisation.  On  the 
other  hand,  the  relative  similarity  of  the  ratings  among  the  participants  serves  as  an  indication  of 
agreement  on  the  value  of  research  on  the  topics  identified  among  the  focus  groups. 

Although  participants  discussed  pertinent  issues  at  different  levels  (e.g.,  individual  and  group), 
there  was  significant  interest  in  the  community-wide,  inter-organizational  issues.  Participants 
acknowledged  that  there  is  a  certain  degree  of  duplication  of  effort  among  different  organizations, 
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which  leads  to  a  suboptimal  use  of  the  available  resources  in  the  community.  In  this  light, 
participants  discussed  such  topics  as  the  need  for  better  information  sharing,  collaboration,  and 
coordination  within  the  community,  the  need  to  promote  joint  projects  and  cross-organizational 
focus  in  sharing  responsibilities.  Another  community-wide  issue,  professionalization  of  intelligence 
analysis,  also  received  significant  attention  in  all  four  focus  groups.  Discussions  focused  on  such 
topics  as  standardization  of  product  and  training,  implementation  of  a  community-wide 
secondment  program,  and  establishing  a  community-wide  career  path  for  intelligence  analysts.  The 
quantitative  ratings  of  the  importance  of  analytic  standards  collected  in  Phase  1  showed  that 
participants  viewed  having  standards  as  quite  valuable,  however  not  yet  adequately  in  place. 
Overall,  the  overarching  community  concerns  were  a  high  priority  for  the  members  of  the 
intelligence  community  represented  within  the  focus  groups. 

Other  issues  related  to  the  tools,  techniques  and  practice  of  intelligence  were  also  discussed  at 
length  during  the  sessions.  These  issues  included  the  need  to  document  existing  tools  and  practices, 
as  well  to  evaluate  their  effectiveness.  Communicating  intelligence  through  improving  the 
communication  of  uncertainty,  improving  information  visualization  techniques,  and  allowing  for 
flexibility  in  presentation  formats  to  suit  different  preferences  and  styles  were  also  identified  as 
priorities.  On  the  other  hand,  participants  also  noted  the  need  for  standardization  of  classification 
and  language  used  to  communicate  risk  and  uncertainty. 

The  relationship  with  consumers  was  another  topic  that  received  considerable  attention  in  all  focus 
groups.  Some  participants  observed  that  intelligence  consumers  (especially  when  they  are  removed 
from  the  intelligence  producers)  often  lack  the  understanding  of  the  intelligence  process,  how  to 
properly  formulate  their  intelligence  requirements,  and  how  to  interpret  information  in  intelligence 
products.  Hence,  the  need  for  educating  consumers  about  intelligence  was  discussed.  The  lack  of 
feedback  from  consumers  on  intelligence  products  was  also  raised  as  a  significant  issue  for  many 
organizations.  The  above-mentioned  consumer-producer  relationship  issues  are  not  prevalent  to  all 
organizations,  however.  The  consumer  base  for  different  organizations  within  the  community 
varies  considerably,  which  affects  the  predominance  of  the  various  producer-consumer  relationship 
issues  on  their  operations. 

Issues  related  to  analyst  recruitment,  motivation,  development,  and  performance  evaluation  were 
also  discussed.  Participants  discussed  such  issues  as  the  need  for  identifying  the  essential  skills 
required  of  analysts  that  could  be  incorporated  into  the  selection  process  and  the  need  to  improve 
the  selection  process  itself.  Some  participants  indicated  that  the  current  intelligence  analyst 
selection  criteria  and  processes  are  too  general  and  do  not  always  allow  for  effective  screening  of 
candidates.  This  issue  is  related  to  the  need  of  identifying  the  essential  skills  required  of 
intelligence  analysts  and  developing  assessment  tools  that  would  allow  evaluating  these  skills. 
Participants  commented  that  retention  of  analysts  has  been  a  considerable  issue  for  many 
organizations  in  light  of  limited  career  progression  available  for  analysts.  The  increasing 
expectations  of  the  new  generation  of  recruits  that  some  participants  observed  further  exacerbate 
this  issue. 

Participants  also  highlighted  the  existing  challenges  in  evaluating  performance  of  intelligence 
analysts  and  quality  of  intelligence  products.  Several  potential  performance  indicators  were 
mentioned,  including  the  predictive  accuracy  of  a  report  and  its  impact,  and  the  product’s  ability  to 
provide  timely  information  to  the  consumer.  Participants  suggested  that  tracking  the  actual  usage  of 
the  products  and  developing  reviewable  processes  with  audit  trails  would  facilitate  the  performance 
evaluation  process.  However,  in  many  organizations  currently  there  are  no  clear  performance 
expectations,  which  are  linked  to  objective  indicators,  as  well  as  there  is  no  consensus  within  the 
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community  on  what  performance  indicators  are  the  most  critical.  There  was  a  consensus  among  the 
participants  on  the  relative  importance  of  evaluating  intelligence  process  versus  intelligence 
product.  As  was  indicated  by  the  quantitative  evaluations  collected  during  the  Phase  1,  participants 
view  both  process  and  product  evaluations  as  equally  important,  which  suggests  that  there  is  a  need 
for  both  product-  and  process-base  performance  evaluation  indicators. 

The  development  of  performance  evaluation  measures  also  has  an  impact  on  the  ability  to  identify 
specific  selection  criteria  for  analysts.  In  order  to  identify  critical  skills  for  analysis,  specific  skills 
and  stable  individual  differences  have  to  be  first  linked  with  analysts’  performance,  which  requires 
sound  performance  evaluation  criteria  and  measures. 

Overall,  the  focus  group  discussions  indicated  a  wide  range  of  issues  and  challenges  facing  the 
intelligence  community,  as  well  as  offering  considerable  insight  about  how  some  of  these  issues 
can  be  addressed  in  the  future.  This  issue  is  addressed  in  more  detail  in  the  remaining  chapter. 


3.4  Limitations 

This  research  has  some  limitations  that  should  be  considered  in  interpreting  its  results. 

The  list  of  issues  identified  by  the  participants  is  not  a  complete  and  exhaustive  list  of  capability 
challenges  faced  by  the  community.  The  first  two  focus  groups  used  a  “brainstorming”  process  to 
allow  participants  to  identify  the  topics  and  issues  of  most  concern  from  their  own  perspectives. 
Although  this  free-flowing  environment  promoted  good  discussion,  it  does  not  guarantee  that  all  of 
the  pertinent  issues  were  identified  or  received  sufficient  attention  during  the  discussion.  The  Phase 
2  focus  groups  validated  the  set  of  issues  identified  during  Phase  1  and  expanded  it  to  a  certain 
extent.  However,  Phase  2  discussions  were  guided  by  the  issues  identified  during  the  Phase  1 , 
which  had  an  influence  on  the  direction  the  discussions  took  and  might  have  prevented  other  topics 
from  emerging. 

It  is  also  worth  noting  that  participants  represented  in  this  convenience  sample  may  or  may  not  be 
representative  of  community  members  as  a  whole.  This  also  suggests  that  there  may  be  additional 
issues  or  challenges  within  the  community  that  were  not  identified  by  the  current  group  of 
participants,  or  that  issues  may  be  weighted  differently  within  a  larger  sample.  Some  of  these  may 
be  identified  in  the  planned  community-wide  survey. 

Another  limitation  is  that  the  differences  in  quantitative  assessment  methodology  from  Phase  1  to 
Phase  2  prevent  aggregation  of  the  quantitative  data.  As  the  two  phases  of  research  used  a  different 
format,  it  is  not  possible  to  quantitatively  compare  the  value  ratings  of  issues  collected  from  Phase 
1  and  Phase  2.  Similarly,  the  quantitative  evaluations  cannot  be  used  to  prioritize  the  issues 
identified  as  was  initially  hoped.  However,  the  qualitative  data  provide  an  important  bridge 
between  the  two  phases  of  research. 
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4.  Research  Implications  and  Conclusion 

4.1  Implications  for  Research 

As  one  of  the  primary  goals  of  this  research  was  to  help  inform  future  research  efforts,  this  section 
identifies  the  issues  and  challenges  identified  within  these  focus  groups  that  can  be  supported  by 
future  research  from  the  behavioural  science  and  other  domains.  Table  3  lists  different  issues 
discussed  in  the  focus  groups  along  with  the  proposed  research  directions  that  either  emerged 
directly  from  the  focus  group  discussions  or  were  indirectly  implied. 


Table  3.  Research  in  support  of  the  intelligence  community 


Broad  category 

Specific  issues  and  topics 

Research  supporting  the  intelligence  community 

1)  Analyst 
recruitment, 
motivation  and 
career  path 

1.1  Identifying  analyst  selection 
criteria  and  developing  selection 
tools 

-  Identify  essential  skills  and  abilities  for  intelligence  analysis; 

-  Create  analyst  selection  criteria  using  cognitive  testing; 

-  Develop  and  validate  selection  tools  in  relation  to  objective 
measures  of  performance; 

1.2  Identifying  drivers  of  analyst 
motivation  for  recruitment, 
retention,  and  career  progression 

-  Identify  and  analyze  intelligence  analyst  motivational  drivers; 

-  Identify  current  and  develop  potential  organizational  strategies 
to  strengthen  analyst  motivational  drivers  to  facilitate  analyst 
retention; 

-  Analyze  currently  available  career  paths  and  develop  potential 
career  paths  to  promote  better  career  progression  for  analysts; 

2)  Analysis  of 
tasks  and 
competencies 

2.1  Conducting  task  analysis  for 
different  roles  (e.g.,  analyst, 
manager,  collector) 

-  Use  cognitive  task  analysis  to  understand  requirements  of 
personnel  in  different  roles; 

2.2  Identifying  competencies  for 
different  roles  and  tasks  (e.g., 
communication) 

-  Identify,  validate  and  create  tools  for  identifying  core 
competencies  for  intelligence  personnel  in  varying  positions; 

3)  Performance 
Evaluation 

3.1  Defining  expectations  of 
analysts'  performance 

-  Use  research  to  support  the  performance  evaluation  process 
for  intelligence  personnel; 

-  Help  to  identify  and  quantify  appropriate  performance  metrics; 

-  Develop  methods  and  tools  that  would  allow  tracking  analysts’ 
performance  over  time  throughout  their  career; 

3.2  Developing  objective 
evaluation  methods  of  process 
and  products 

-  Create  and  validate  tools  for  understanding  the  value  and 
knowledge  transfer  of  intelligence  products  to  the  consumer; 

-  Use  research  to  help  quantify  the  objective  value  of  an 
intelligence  product  and  identify  the  critical  evaluative 
dimensions; 

-  Evaluate  the  performance  of  an  intelligence  unit  against 
performance  criteria; 

3.3  -  Developing  tools  to  support 
performance  evaluation  and  audit 
trails 

-  Create  and  validate  tools  for  the  documenting  the  usage  of 
intelligence  products; 

-  Create  and  validate  tools  to  identify  audit  trails  for  intelligence 
products  and  illustrate  decision-making  processes; 

4)  Mentoring 
and  Feedback 

4.1  Identifying  effective  methods 
for  the  mentoring  of  analysts; 

-  Identify  strategies  that  promote  knowledge  transfer  among 
analysts; 

-  Conduct  research  to  critically  assess  the  impact  of  mentoring 
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Broad  category 

Specific  issues  and  topics 

Research  supporting  the  intelligence  community 

in  relation  to  key  outcomes  (e.g.,  performance,  retention, 
engagement) 

4.2  Instilling  openness  and 
acceptance  to  feedback  and 
criticism  (and  understanding  the 
precursors  of  the  lack  of 
openness  to  feedback). 

-  Conduct  research  to  understand  the  precursors  to  lack  of 
openness  to  feedback; 

-  Develop  intervention  measures  and  training  techniques  to 
promote  greater  openness  to  criticism  and  feedback; 

5)  Producer/ 

Consumer 

Relationship 

5.1  Clarifying  the  optimal 
producer-consumer  relationship 
(and  the  manager's  role  in  this 
process) 

-  Analyze  socio-technical  environment  of  the  producer- 
consumer  interactions  and  requirements; 

-  Identify  ways  to  overcome  knowledge  and  experiential  barriers 
between  producers  and  consumers  of  intelligence; 

-  Document  the  varied  nature  of  the  producer/consumer 
relationship  within  Canada,  identifying  best  practices; 

-  Research  characteristics  of  the  consumers; 

5.2  Clarifying  the  analyst's  role  in 
understanding  and  defining 
product  requirements 

-  Analyze  the  nature  of  the  producer-consumer  relationship  and 
the  nature  of  consumer  requirements; 

-  Explore  the  effectiveness  of  varying  communication  strategies 
on  analyst/consumer  relationships; 

5.3  Educating  consumers  about 
intelligence 

-  Develop  educational  strategies  and  facilitate  the  consumer 
education  process; 

5.4  Identifying  how  products 
impact  decision-makers 

-  Identify  the  key  factors  that  determine  impact  of  intelligence 
products  on  decision  makers; 

-  Develop  measures  to  evaluate  these  key  factors  and  the 
impact  of  intelligence  products  on  decision-makers; 

6)  Analyst- 

Collector 

Understanding 

6.1  Promoting  a  clear 
understanding  of  roles  and 
interdependencies  of  analysts 
and  collectors 

-  Research  the  nature  and  constraints  of  the  analyst/collector 
relationship  within  varying  segments  of  the  community  (e.g., 
strategic/tactical); 

7)  Collegial 
Collaboration 

7.1  Determining  effective 
structures  for  analytic  teams; 

-  Research  the  most  effective  structure  for  analytic  teams; 

7.2  -  Instilling  a  collaborative  spirit 
and  promoting  informal  networks. 

-  Research  ways  to  promote  and  support  informal  networks 
through  organizational  means  that  enable  collaboration; 

8) 

Communication 
of  Intelligence 

8.1  Understanding  and  improving 
the  communication  of  uncertainty 

-  Research  ways  to  improve  the  communication  of  uncertainty 
and  risk; 

8.2  Utilizing  information 
visualization  techniques  with 
consumers 

-  Develop  and  validate  new  information  presentation  techniques 
for  communicating  with  consumers; 

-  Investigate  how  to  best  tailor  information  presentation  to  the 
preferences  and  needs  of  consumers; 

8.3  Impact  of  existing  formats  and 
varying  communication  of  reports 

-  Evaluate  the  effectiveness  of  existing  report  formats  on 
communication  of  intelligence; 

-  Develop  and  evaluate  alternative  report  formats; 

9)  Tools, 
Techniques 
and  Practices 
(TTPs) 

9.1  Documenting  and  evaluating 
current  TTPs  and  developing  new 
TTPs. 

-  Document  and  validate  existing  TTPs; 

-  Determine  realistic  measures  of  performance  in  preparation 
for  validation  efforts; 

-  Identify  discrepancies  between  prescribed  approaches  (e.g.,  in 
doctrine)  and  actual  practice  within  the  intelligence  community; 

-  Identify  the  gaps  in  the  existing  TTPs,  and  develop  and 
validate  new  TTPs; 

-  Evaluate  congruence  between  doctrine  and  human 
capabilities  (e.g.,  judgment  and  decision-making); 
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Broad  category 

Specific  issues  and  topics 

Research  supporting  the  intelligence  community 

-  Create  a  common  resource  hub  for  tool  support  and  training 
(document  personnel  trained  in  specific  techniques  who  could 
serve  as  resources); 

-  Establish  technical  support  service  to  support  intelligence 
managers  in  evaluating  the  usability  and  feasibility  of  new 
technologies; 

10) 

Intelligence 

Community 

10.1  -  Understanding  constraints 
and  barriers  to  knowledge  use 
imposed  by  organizational 
structure 

-  Map  information  flow  and  knowledge  use  within  different 
organizations  and  identify  constraints  and  barriers  to 
knowledge  and  information  flow  due  to  various  factors, 
including  organizational  size,  complexity,  and  structure; 

10.2  -  Better  coordination, 
collaboration,  and  information 
sharing  among  agencies 
and 

10.2.1  -  Facilitating  collaboration 
within  the  community 

-  Document  bottlenecks  in  the  flow  of  information  within  the 
intelligence  community; 

-  Identify  reasons  for  failures  of  collaboration  (e.g.,  lack  of 
willingness  or  lack  of  collaborative  tools); 

-  Research  collaborative  tools  that  would  benefit  the  intelligence 
community  (e.g.  “dashboard”  that  displays  the  ongoing  efforts 
within  the  community); 

-  Conduct  social  network  analysis  research  of  the  intelligence 
community  to  identify  any  gaps  in  collaboration; 

10.2.2  -  Developing  a  resource 
sharing  mechanism 

-  Document  the  available  technical  and  non-technical  resources 
for  information  sharing  within  organizations  in  the  community; 

-  Research  the  merits  of  available  collaborative  tools,  such  as 
shared  websites  or  portals  for  the  sharing  of  documents,  and 
identify  or  develop  a  viable  resource  sharing  mechanism; 

10.2.3  -  Developing  standards  for 
different  information  sources 

-  Identify  existing  information  sources  and  their  characteristics; 

-  Identify  current  usage  requirements  that  different 
organizations  have  for  different  information  sources; 

-  Support  the  development  of  standards  for  varying  sources  of 
intelligence  information; 

10.3  -  Cross-organizational 
review  of  doctrine,  tools, 
techniques  and  practices 

-  Document  cross-agency  doctrine  and  TTPs  with  a  view  to 
identifying  similarities  and  differences; 

10.4  -  Professionalization  of 
intelligence  analysis 

-  Research  the  intelligence  analyst  identity  and  professional 
characteristics; 

-  Document  the  “intelligence  landscape”  and  training 
requirements  in  the  community; 

-  Support  the  development  and  implementation  of  training 
curriculum  and  standardization  of  the  intelligence  training; 

-  Research  the  impact  of  different  types  of  training; 

10.4.1  -  Developing  common 
standards 

-  Identify  potential  areas  that  might  benefit  from  standardization; 

-  Support  the  development  of  standardization  practices  and 
standard  implementation  procedures; 

-  Support  the  standardization  of  the  intelligence  process; 

-  Analyze  the  potential  impacts  of  increased  standardization  and 
formalization  on  the  profession; 

10.4.2  -  Developing  community¬ 
wide  career  path  for  analysts 

-  Research  ways  to  improve  career  paths  for  analysts; 

-  Support  the  creation  and  coordination  of  a  secondment 
initiative  within  the  community; 

-  Analyse  benefits,  drawbacks,  and  organizational  impact  of 
secondments. 
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There  is  significant  potential  for  research  to  make  a  meaningful  contribution  to  improving 
intelligence  production  in  the  future.  Many  of  the  research  issues  identified  in  Table  3  show  the 
potential  contributions  of  behavioural  science  to  help  better  mesh  human  capabilities  with  the 
requirements  of  intelligence  production  (e.g.,  understanding  judgement  and  decision-making 
processes  that  might  influence  analysts).  Given  the  full  range  of  possible  issues  to  address,  future 
efforts  should  work  to  further  refine  research  topics  and  to  identify  the  most  critical  issues  for  the 
intelligence  community. 


4.2  Conclusion 

The  aim  of  the  research  described  in  this  report  was  to  capture  a  broader  range  of  perspectives  from 
the  Canadian  intelligence  community.  Focus  group  discussions  with  members  of  the  intelligence 
community  offered  rich  insights  into  the  current  state  of  affairs  within  the  community.  Focus  group 
participants  seemed  genuinely  engaged  in  the  process  of  discussing  both  strengths  and  challenges 
and  seemed  motivated  to  offer  candid  observations  about  the  intelligence  community  in  order  to 
explore  how  current  challenges  can  be  addressed  through  future  research.  At  the  same  time,  focus 
group  discussions  consistently  suggested  that  members  of  the  intelligence  community  in  Canada 
function  quite  well  to  produce  quality  intelligence  products  despite  the  time  pressures  and 
constraints  that  they  typically  face. 

Although  current  functioning  was  perceived  to  be  adequate,  the  intelligence  community  in  Canada 
faces  internal  pressure  to  evolve  and  appears  to  be  motivated  to  push  itself  forward,  and  to 
strengthen  its  practices  at  a  broad  level.  Participants  within  the  focus  groups  expressed  both  great 
pride  in  their  work  and  in  their  community,  while  simultaneously  expressing  eagerness  to  further 
develop  their  enterprise.  With  years  of  accumulated  experience  among  its  members,  the  focus 
group  participants  identified  a  number  of  areas  for  potential  improvement  in  order  to  advance  the 
interests  of  the  community  and  further  improve  the  internal  systems  within  organizations. 

Even  though  there  was  considerable  range  of  experience  represented  within  the  focus  groups,  there 
was  a  good  deal  of  agreement  about  the  key  challenges.  Perhaps  not  surprisingly,  discussions 
indicated  a  high  level  of  interrelatedness  among  the  challenges  facing  the  community.  For  example, 
the  current  lack  of  collaborative  tools  within  the  intelligence  community  is  inextricably  linked  with 
the  ability  to  share  information  and  to  adopt  more  standardized  approaches  among  organizations. 
Unless  the  many  different  agencies  that  must  work  collaboratively  have  the  resources  and  tools  that 
they  need  (e.g.,  common  or  compatible  software),  this  issue  will  not  be  adequately  addressed  and 
infonnation  sharing  will  not  be  at  an  optimal  level. 

Participating  intelligence  professionals  expressed  considerable  interest  in  increased 
professionalization  of  the  area,  standardization  of  intelligence  processes  and  products,  and  a  desire 
to  cultivate  a  more  community-wide  perspective.  However,  a  common  theme  throughout  the  focus 
groups  was  that  although  seeking  additional  standardization  and  consistency  of  approaches  within 
the  community  is  important,  it  is  also  important  to  recognize  the  uniqueness  of  the  roles  played 
(and  the  demands  on)  the  many  players  within  the  community.  As  noted  earlier,  participants 
strongly  suggested  that  a  “one  size  fits  all”  approach  would  not  benefit  the  community  in  the  long 
run. 

It  is  also  important  to  note  that  many  of  the  problems  identified  by  personnel  within  the  focus 
groups  are  not  necessarily  unique  to  the  intelligence  community.  For  example,  participants  noted 
that  human  resource  issues  (e.g.,  retention  and  career  paths)  are  common  challenges  within  many 
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organizations.  Similarly,  within  any  system  in  which  multiple  complex  organizations  must  share 
information  and  coordinate  their  efforts,  the  need  for  improved  collaboration  seems  likely  to  be 
identified  as  a  familiar  challenge.  This  suggests  that  although  it  is  important  to  address  the 
concerns  identified,  it  may  be  helpful  to  consider  how  other  organizations  with  different  mandates 
cope  with  similar  concerns. 

The  results  of  this  research  are  congruent  with  the  interview  findings  of  Derbentseva,  McLellan 
and  Mandel  (2011).  Although  this  work  represents  a  broad  range  of  experience  from  many 
additional  organizations,  many  of  the  themes  emerging  within  these  focus  groups  were  also 
identified  by  the  previous  study  (e.g.,  the  need  to  identify  essential  competences,  evaluation  of 
tools  and  techniques  and  development  of  new  products).  Although  this  research  explores  and 
elaborates  a  broader  range  of  issues,  it  is  reassuring  to  see  many  similar  issues  identified  by  other 
members  of  the  intelligence  community. 

Focus  group  discussions  showed  high  levels  of  synergy  between  the  current  needs  of  the 
intelligence  community  and  the  potential  for  scientific  research  to  contribute  to  documenting, 
evaluating  and  developing  effective  products  and  processes.  Members  of  the  intelligence 
community  commonly  referenced  the  need  for  empirically  validated  processes,  techniques  and 
outcomes  in  order  to  ensure  the  highest  possible  levels  of  effectiveness  within  their  field.  It  is 
encouraging  that  personnel  within  the  intelligence  community  saw  research  as  being  a  valuable 
way  to  address  the  current  challenges  explored  during  discussions.  This  suggests  that  scientific 
approaches  are  likely  to  offer  significant  value  to  the  community  as  it  works  to  better  formalize  the 
production  of  intelligence  analysis. 

At  the  same  time,  however,  discussions  also  highlighted  the  need  for  continued  discourse  to  help 
bridge  communication  and  experiential  gaps  between  intelligence  practitioners  and  scientists.  Just 
as  focus  group  participants  indicated  that  analysts  and  consumers  often  speak  a  somewhat  different 
language,  similar  challenges  in  bridging  the  gap  between  practitioners  and  researchers  were  evident 
during  the  focus  groups  and  have  been  identified  as  a  challenge  for  effective  science-practitioner 
partnerships  in  the  intelligence  domain  (Mandel,  2009;  in  press).  Practitioners  varied  in  their 
ability  to  see  and  articulate  how  research  could  inform  their  efforts,  with  some  personnel  seeming 
very  able  to  express  natural  linkages  between  their  practice  and  research,  and  others  clearly 
identifying  that  research  might  be  able  to  assist  them,  but  indicating  that  they  had  not  been 
adequately  exposed  to  research  to  understand  exactly  how.  Similarly,  due  to  their  limited  exposure 
to  the  day-to-day  challenges  of  intelligence  personnel,  researchers  conducting  the  focus  groups 
might  not  have  fully  grasped  all  of  the  nuanced  needs  of  the  intelligence  community  that  future 
research  could  address.  One  potential  solution  to  bridging  this  gap  between  practitioners  and 
researchers  would  be  to  establish  a  practitioner-researcher  collaboration  network  that  would 
provide  a  venue  for  continuous  dialogue  and  the  development  of  mutual  awareness  and 
understanding.  Furthermore,  the  interaction  and  dialogue  to  more  clearly  define  the  issues  and 
challenges  within  the  intelligence  community  will  need  to  continue. 
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Annex  A- Additional  questions  posed  to 
Phase  1  participants 

There  were  eight  additional  specific  questions  posed  to  participants  of  the  first  two  focus  groups. 
For  each  question,  participants  were  instructed  to  select  one  of  the  available  answer  options.  Both 
the  questions  and  their  available  answer  options  are  presented  below: 

1 .  Flave  you  thought  about  the  process-outcome  distinction  in  assessing  the  quality  of  analytic 
products  and  the  performance  of  analysts?  Answer  options: 

o  YES 
o  NO 

2.  Flow  would  you  rate  the  relative  importance  of  process  and  outcome  evaluations?  Answer 
options 

1.  ALL  PROCESS 

2.  MOSTLY  PROCESS,  A  BIT  OF  OUTCOME 

3.  EQUALLY  BALANCED 

4.  MOSTLY  OUTCOME,  A  BIT  OF  PROCESS 

5.  ALL  OUTCOME 

3.  Flow  important  it  is  to  have  analytic  standards  in  your  organization? 

4.  Flow  important  it  is  to  have  analytic  standards  in  the  community? 

The  above  two  questions  were  rated  on  the  same  5-point  scale  ranging  from  1  (of  little  or  no 
value)  to  5  (extremely  valuable). 

5.  To  what  extent  do  you  think  these  standards  are  adequately  in  place  in  your  organization? 

6.  To  what  extent  do  you  think  these  standards  are  adequately  in  place  in  the  community? 

The  above  two  questions  had  the  following  answer  options: 

1.  Not  at  all 

2.  About  %  of  the  way  there 

3.  About  Vi  of  the  way  there 

4.  About  ki  of  the  way  there 

5.  All  the  way  there 

7.  Flow  valuable  do  you  think  behavioral  science  research  could  be  in  helping  to  improve 
standards?  On  a  5-point  scale  ranging  from  1  (of  little  or  no  value)  to  5  (extremely 
valuable) 

8.  How  would  you  rate  the  relative  importance  of  description  and  prediction  in  intelligence 
analysis?  Answer  options  included: 

1 .  Description  is  important,  prediction  is  not 

2.  Description  is  more  important  than  prediction 

3.  Description  and  prediction  are  equally  important 

4.  Prediction  is  more  important  than  description 

5.  Prediction  is  important,  description  is  not 
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Annex  B  -Data  Reduction  Process 


Table  4  shows  how  the  issues  and  topics  generated  in  Phase  1  (Focus  Groups  1  and  2)  were 
mapped  into  the  broad  categories  and  issues  used  during  Phase  2. 


Table  4.  Aggregation  of  issues  generated  during  the  Phase  1  into  broad  categories  used  during  Phase  2 


Broad  category 

Specific  topic/issue 

Issues  identified  in  Focus  Groups  1  and  2 
(SI  and  S2  respectively) 

1)  Analyst 
recruitment, 
motivation  and 
career  path 

1.1  -  Identifying  analyst 
selection  criteria  and 
developing  selection  tools; 

S2-  Selection  tools/job  profiles 

1.2  -  Identifying  drivers  of 
analyst  motivation  for 
recruitment,  retention,  and 
career  progression. 

SI- Analyst  incentives  -  Recruitment 

SI- Analyst  incentives  -  Retention 

SI- Career  paths 

SI- Retention 

51-  Analyst/manager  progression 

52-  Drivers  of  analyst  motivation 

2)  Analysis  of  tasks 
and  competencies 

2.1  -  Conducting  task 
analysis  for  different  roles 
(e.g.,  analyst,  manager, 
collector); 

S2-  Task  analysis  for  collectors 

S2-  Task  analysis  for  analysts 

2.2  -  Identifying 
competencies  for  different 
roles  and  tasks  (e.g., 
communication). 

SI-  Competencies  for  different  roles 

SI-  Competencies  for  analysis 

SI-  Competencies  for  communication 

3)  Performance 
Evaluation 

3.1  -  Defining  expectations  of 
analysts'  performance; 

S2-  Defining  experience  and  expectations 

3.2  -  Developing  objective 
evaluation  methods  of 
process  and  products; 

S2-  Objective  evaluation  of 
process/products 

3.3  -  Developing  tools  to 
support  performance 
evaluation  and  audit  trails. 

S2-  Tracking  of  actual  performance  after 
selection 

51-  Tools  to  support  audit  trails 

52-  Providing  tools  to  support  evaluation 

4)  Mentoring  and 
Feedback 

4.1  -  Identifying  effective 
methods  for  the  mentoring  of 
analysts; 

SI- Mentoring 

4.2  -  Instilling  openness  and 
acceptance  to  feedback  and 
criticism  (and  understanding 
the  precursors  of  the  lack  of 
openness  to  feedback). 

SI- Accepting  feedback/criticism 

SI-  Openness  to  feedback/legitimacy  of 
authority 

5) 

Producer/Consumer 

Relationship 

5.1  -  Clarifying  the  optimal 
producer-consumer 
relationship  (and  the 
manager's  role  in  this 
process); 

SI-  Optimal  analyst/consumer 
relationship 

SI-  Producer  vs.  consumer 

SI-  Manager  to  clarify  role  of  consumer 

SI-  Producer/consumer  synergy 

5.2  -  Clarifying  the  analyst's 

SI-  Producer/consumer  synergy 
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Broad  category 

Specific  topic/issue 

Issues  identified  in  Focus  Groups  1  and  2 
(SI  and  S2  respectively) 

role  in  understanding  and 
defining  product 
requirements; 

SI-  Understanding  consumer 
requirements 

SI-  Driving  consumer  requirements 

5.3  -  Educating  consumers 
about  intelligence; 

SI-  Consumer  education 

SI-  Educating  consumers  about 
intelligence 

5.4  -  Identifying  how  products 
impact  decision-makers. 

S2-  Track  usage  of  products 

51-  Presentation  of  indications  and 
warnings 

52-  Flow  IA  impacts  decision-making 

6)  Analyst-Collector 
Understanding  and 
Terms  of  Reference 

SI-  Understanding  consumer 
requirements 

SI-  Terms  of  reference  -  operator/analyst 

7)  Collegial 
Collaboration 

7.1  -  Determining  effective 
structures  for  analytic  teams; 

SI-  Best  structure  for  analytic  teams 

7.2  -  Instilling  a  collaborative 
spirit  and  promoting  informal 
networks. 

SI-  Instill  collaborative  spirit 

SI- Promoting  informal  networks 

8)  Communication 
of  Intelligence 

8.1  -  Understanding  and 
improving  the  communication 
of  uncertainty 

SI-  Presentation  of  indications  and 
warnings 

8.2  -  Utilizing  information 
visualization  techniques  with 
consumers 

S2-  Information  visualization  techniques  - 
consumers 

8.3-  Impact  of  existing 
formats  and  varying 
communication  of  reports 

S2-  Impact  of  existing  formats  (e.g., 
layout) 

S2-  Impact  of  varying  communication 

SI- Format  of  reports 

9)  Tools, 

Techniques  and 
Practices  (TTPs) 

9.1  -  Documenting  and 
evaluating  current  TTPs  and 
developing  new  TTPs. 

S2-  Methodological 
approaches/techniques 

S2-  Evaluating  methodologies 

S2-  New  tools  and  techniques 

S2-  Information  visualization  techniques  - 
Analysis 

S2-  Validating  doctrine,  policies,  TTPs 
(human  issues) 

S2-  Explore  bottlenecks  in 
policy/government  systems 

10)  Intelligence 
Community 

10.1  -  Understanding 
constraints  and  barriers  to 
knowledge  use  imposed  by 
organizational  structure 

51-  Need  for  information  sharing 

52-  Develop  information  management 

TTPs 

51-  Org  structures  present  barriers  to  use 
of  knowledge 

52-  Identifying  bottlenecks 

10.2 -Better  coordination, 
collaboration,  and  information 
sharing  among  agencies: 

SI-  Better  coordination  and  focus 

SI-  Dialogue  between  academics  & 
practitioners 

SI-  Divergent  ethos/practice  among  orgs 
SI-  Need  for  information  sharing 

10.2.1  -  Facilitating 
collaboration  within  the 

SI-  Effectiveness  of  varying  entry  paths 
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Broad  category 

Specific  topic/issue 

Issues  identified  in  Focus  Groups  1  and  2 
(SI  and  S2  respectively) 

community; 

10.2.2  -  Developing  a 
resource  sharing  mechanism; 

S2-  Resource  sharing  mechanism 

10.2.3  -  Developing 
standards  for  different 
information  sources. 

S2-  Developing  standards  for  different 
information  sources 

10.3  -  Cross-organizational 
review  of  doctrine,  tools, 
techniques  and  practices. 

SI-  Intra  versus  interagency  collaboration 

51-  Most  effective  structure/model 

52-  Cross-org  review  -  doctrine  and 

TTPs 

10.4  -  Professionalization  of 
intelligence  analysis: 

SI-  Professionalization  of  field 

SI-  Common  language  for 
communication 

10.4.1  -  Developing  common 
standards; 

SI- Common  standards 

SI- Common  standards 

10.4.2  -  Developing 
community-wide  career  path 
for  analysts 

S2-  Identifying  bottlenecks 

51-  Effectiveness  of  varying  entry  paths 

52-  Resource  sharing  mechanism 
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Annex  C  -  The  original  list  of  issues  identified 
and  rated  by  the  Phase  1  participants 


Table  5  shows  the  issues  identified  by  the  Phase  1  focus  groups  during  the  discussion  of  topics 
related  to  individual,  group,  organizational,  and  community  levels.  The  table  contains  a  brief 
description  of  how  focus  group  members  described  the  issue2,  and  the  mean  importance  of  each 
issue  (as  rated  on  the  5 -point  scale).  Participants  identified  a  total  of  44  issues  when  exploring  the 
intelligence  process  at  the  hierarchical  level,  with  ten  issues  at  the  individual  level,  six  at  the  group 
level,  12  at  the  organizational  level  and  16  at  the  community  level. 


Table  5.  Issues  generated  at  the  hierarchical  level  of  organization  during  the  Phase  1  and  their  mean 
ratings 


Level 

Issue 

Description  of  the  issue  by  participants 

FG 

Mean 

rating 

Individual 

Help  consumers  define 
requirements 

Consumers  do  not  know  how  to  define  exactly 
what  they  want 

1 

4.7 

Analyst/collector  understanding 

Analyst  and  collectors  need  to  understand 
each  other 

1 

4.2 

Analyst  incentives  - 
Recruitment 

Understanding  the  motivation  to  be  an  analyst 

1 

3.7 

Analyst  incentives  -  Retention 

Understanding  the  motivation  to  remain  an 
analyst 

1 

4.2 

Competencies  for  different  roles 

Competencies  required  to  be  an  analyst, 
manager  etc. 

1 

3.8 

Competencies  for  analysis 

Identify  the  critical  skills  necessary  for  being 
an  analyst 

2 

3.7 

Competencies  for 
communication 

Identify  critical  communication  skills  necessary 
for  working  in  the  intelligence  community 

2 

3.3 

Instil  collaborative  spirit 

Increase  collaborative  spirit  at  individual  level 

2 

3.6 

Accepting  feedback/criticism 

Facilitate  acceptance  of  feedback  and 
criticism  be  improved 

2 

3.1 

Promoting  informal  networks 

Identify  ways  to  promote  and  support  good 
informal  networks 

2 

3.3 

Group 

Optimal  analyst/consumer 
relationship 

Understanding  how  to  facilitate  the  most 
productive  analyst/consumer  relationship 

1 

3.7 

Producer  vs.  consumer 

Producers  and  consumer  relationship 

1 

3.3 

Best  structure  for  analytic 
teams 

Best  structure  of  analytic  teams  (e.g.,  all  one 
geographic  area,  etc.) 

1 

4.2 

;For  full  discussion  of  the  topics  that  emerged  in  the  focus  groups,  see  qualitative  results  in  Section  3.2. 
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Level 

Issue 

Description  of  the  issue  by  participants 

FG 

Mean 

rating 

Manager  to  clarify  role  of 
consumer 

What  is  the  manager's  role  in  interacting  with 
the  consumer? 

2 

3.4 

Mentoring 

How  can  mentoring  benefit  the  intelligence 
community? 

2 

3.6 

Openness  to 

feedback/legitimacy  of  authority 

How  can  openness  to  feedback  and 
recognition  of  authority  be  increased? 

2 

3.0 

Organization 

Defining  intelligence  landscape 
in  2011 

How  should/will  intelligence  analysis  be 
defined  in  the  future? 

1 

4.2 

Career  paths 

Lack  of  career  paths  for  analysts  was 
identified  as  a  particular  problem  -  not  all 
analysts  can  be  managers 

1 

3.8 

Divergent  ethos/practice  among 
organizations 

Lack  of  consistency  among  different 
organizations  and  different  cultures  may 
hinder  collaboration 

1 

3.0 

Impact  of  organizational 
structure 

Organizational  structure  can  impact  on  how 
intelligence  analysis  is  produced 

1 

4.0 

Retention 

How  can  analysts  be  retained  over  time? 

1 

3.7 

Analyst/manager  progression 

How  do  analysts  progress  to  become 
managers? 

1 

3.0 

Format  of  reports 

What  format  is  best  for  intelligence  reports? 

2 

3.3 

Producer/consumer  synergy 

How  can  producers  and  consumers  work 
together  better? 

2 

3.7 

Understanding  consumer 
requirements 

How  can  the  requirements  of  consumers  be 
better  understand? 

2 

4.3 

Terms  of  reference  - 
operator/analyst 

Better  clarity  is  required  around  the  roles  of 
operators  and  analysts 

2 

3.6 

Driving  consumer  requirements 

Can  consumer  requirements  be  better  guided 
and  shaped  by  the  1C? 

2 

3.3 

Organizational  structures 
present  barriers  to  use  of 
knowledge 

Can  barriers  resulting  from  organizational 
structure  be  overcome? 

2 

3.4 

Community 

Consumer  education 

How  can  consumers  be  educated  to  ask  their 
questions  more  effectively  and  efficiently? 

1 

4.3 

Impact  of  turnover 

How  can  negative  impacts  of  turnover 
(particularly  of  analysts)  be  mitigated? 

1 

4.0 

Professionalization  of  field 

How  can  intelligence  analysis  be 
professionalized? 

1 

4.3 

Better  coordination  and  focus 

Current  problems  are  redundant  efforts  among 
organizations  and  disproportionate  focus  on 
less  critical  issues  (e.g.,  in  the  media  eye) 

1 

4.3 

Most  effective  structure/model 

What  is  the  most  effective  structure  for  an 
analytic  team? 

1 

3.8 

Common  language  for 
communication 

How  can  language  become  less  idiosyncratic 
to  each  organizations 

1 

3.8 

Common  standards 

Common  standards  from  organization  to 
organization  are  required 

1 

3.8 

Effectiveness  of  varying  entry 
paths 

Do  analysts  that  enter  their  profession  from 
different  paths  perform  equally  well  over  time? 

1 

2.5 
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Level 

Issue 

Description  of  the  issue  by  participants 

FG 

Mean 

rating 

Career  path 

Analysts  often  have  a  very  limited  career  path 

1 

4.0 

Tools  to  support  audit  trails 

What  tools  would  support  the  creation  of  clear 
audit  trails? 

2 

3.4 

Common  standards 

Need  common  standards  within  and  among  IA 
organizations 

2 

3.9 

Dialogue  between  academics  & 
practitioners 

Academics  and  practitioners  need  to  be  able 
to  talk  to  each  other  about  intelligence 

2 

3.0 

Educating  consumers  about 
intelligence 

How  can  consumers  be  better  educated? 

2 

4.1 

Presentation  of  indications  and 
warnings 

Can  indications  and  warnings  be  standardized 
to  have  more  consistency  across 
organizations? 

2 

3.6 

Intra  vs.  interagency 
collaboration 

Need  to  have  better  collaboration  at  both 
levels 

2 

3.7 

Need  for  information  sharing 

How  can  information  sharing  be  promoted? 

2 

4.1 

Table  5  shows  that  most  of  the  topics  listed  while  thinking  about  intelligence  analysis  from  a 
hierarchical  perspective  were  rated  above  the  midpoint  of  the  scale,  ranging  between  quite  valuable 
and  very  valuable.  Helping  consumers  define  their  requirements  (mean  of  4.7)  was  rated  the  most 
valuable  research  priority  of  all  the  issues  and  topics,  and  research  aimed  at  understanding  the 
effectiveness  of  varying  entry  paths  was  given  the  lowest  value  with  a  mean  of  2.5. 

It  is  worth  noting  that  participants  identified  a  higher  number  of  issues  at  the  community  level  and 
rated  their  value  the  highest  of  all  four  levels.  Organizational  issues  were  rated  as  slightly  less 
valuable  than  community  issues,  and  individual  and  group  level  issues  were  rated  the  lowest. 
However,  all  issues  identified  were  rated  as  being  either  quite  valuable  or  very  valuable  to  explore 
in  future  research. 

Table  6  summarizes  the  issues  raised  in  the  “functions  of  science”  session.  For  each  issue,  a 
description  is  provided  along  with  the  mean  rating  of  importance. 


Table  6.  Functional  activity  results  from  the  Phase  1 


Level 

Issue 

Description  of  the  issue  by 
participants 

FG 

Mean 

rating 

Documenting 

Methodological 

approaches/techniques 

Document  what  approaches  and 
techniques  are  available  to  the 
analyst 

1 

3.5 

Track  usage  of 
products 

Explore  ways  to  track  how  much  a 
product  has  been  used  (and  by 
whom) 

1 

3.7 

Explore  bottlenecks  in 

policy/government 

system 

Identify  the  current  bottlenecks  in 
policy/government  systems  that 
hinder  analysis 

1 

3.8 

Cross-organizational 
review  -  doctrine  and 
TTPs 

Compare  doctrine  and  TTPs  across 
organizations  in  order  to  improve 
standardization 

2 

3.6 
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Level 

Issue 

Description  of  the  issue  by 
participants 

FG 

Mean 

rating 

Task  analysis  for 
collectors 

Conduct  an  analysis  of  the  tasks 
required  of  collectors 

2 

3.6 

Task  analysis  for 
analysts 

Conduct  an  analysis  of  the  tasks 
required  of  collectors 

2 

3.7 

Identifying  bottlenecks 

Identify  the  current  bottlenecks  in  the 
system  that  diminish  effectiveness 
and  efficiency 

2 

3.6 

Comparing  concept  vs. 
reality 

Document  the  relationship  between 
current  doctrine  and  standards  within 
the  intelligence  community 

2 

3.7 

Evaluating 

Evaluating 

methodologies 

Evaluate  whether  tools  and 
techniques  available  to  the 
intelligence  community  work 
effectively 

1 

4.0 

Defining  experience 
and  expectations  of 
analysts 

Evaluate  the  necessary  level  of 
experience  and  expectations  of 
analysts 

1 

4.0 

Impact  of  existing 
formats  (e.g.,  layout) 

Evaluate  the  impact  of  format  -  are 
some  more  effective  than  others,  and 
for  what  type  of  audience 

1 

3.7 

Impact  of  varying 
communication 

Evaluate  what  form  of  communication 
is  most  effective 

1 

4.0 

Objective  evaluation  of 
process/products 

Enable  objective  evaluation  of 
intelligence  processes  and  products 

1 

4.2 

Tracking  of  actual 
performance  after 
selection 

Track  and  evaluate  the  performance 
(e.g.,  accuracy)  of  specific  analysts 
over  time 

1 

3.5 

Validating  doctrine, 
policies,  TTPs  in 
relation  to  human 
issues 

Compare  the  congruence  between 
current  doctrines/  practices  and 
human  capabilities  and  limitations 

2 

3.9 

How  intelligence 
analysis  impacts 
decision-making 

Understand  the  impact  of  intelligence 
analysis  on  how  people  make 
decisions? 

2 

4.0 

Evaluating  concept  vs. 
reality 

Evaluate  how  well  current  doctrine 
and  standards  are  implemented  within 
the  community 

2 

4.0 

Drivers  of  analyst 
motivation 

What  motivates  analysts  to 
become/stay  analysts 

2 

3.7 

Developing 

New  tools  and 
techniques 

Develop  new  tools  to  support 
intelligence  production 

1 

4.0 

Information 
visualization 
techniques  - 
consumers 

Develop  information  visualization 
techniques  to  facilitate  communication 
with  consumers 

1 

4.0 

Information 
visualization 
techniques  -  Analysis 

Develop  information  visualization 
techniques  to  assist  analysis 

1 

4.0 

Selection  tools/job 
profiles 

Develop  selection  tools  and  job 
profiles  to  promote  better  analyst 

1 

3.8 
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Level 

Issue 

Description  of  the  issue  by 
participants 

FG 

Mean 

rating 

performance 

Facilitate  coordination 
and  communication  in 
community 

How  can  better  coordination  and 
communication  be  facilitated? 

2 

3.9 

Resource  sharing 
mechanism 

Can  resources  be  better  shared? 

2 

4.0 

Developing  standards 
for  different  information 

sources 

Are  there  common  standards  for  what 
is  expected  of  different  information 
sources? 

2 

3.6 

Providing  tools  to 
support  evaluation 

Can  tools  be  developed  to  support  the 
empirical  evaluation  of  intelligence 
process  and  products? 

2 

3.9 

Develop  information 
management  TTPs 

Develop  TTPs  for  how  information  is 
utilized 

2 

3.6 

Participants  identified  26  issues  related  to  functional  activities,  with  eight  involving  documentation, 
ten  related  to  evaluation  processes,  and  nine  related  to  development.  Activities  related  to  the 
evaluation  of  intelligence  processes  and  products  and  development  activities  were  rated  slightly 
higher  than  documenting  processes. 
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